From 7aaafe7520c1a9d3be75f8f31cccfa01d91f94e5 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 11 Sep 2026 03:28:51 +0800 Subject: [PATCH 01/38] docs: add the Elasticsearch upgrade guide for ACP 4.4 Add docs/en/upgrade/: an Upgrade section entry page and the guide for upgrading a cluster that stores logs with Log Storage for Elasticsearch to ACP 4.4, where new data is written to ClickHouse or an OpenSearch 3.7.0 cluster. The guide covers: - connection Secrets for the target storage and the new Kafka service - the new data path created with a PlatformLogForward that carries the log.alauda.io/legacy-es-upgrade annotation - controlled uninstall of the Log Storage for Elasticsearch plugin - optional migration of the retained historical data - cleanup of the retained Elasticsearch PVCs and PVs --- docs/en/upgrade/elasticsearch-upgrade.mdx | 397 ++++++++++++++++++++++ docs/en/upgrade/index.mdx | 19 ++ 2 files changed, 416 insertions(+) create mode 100644 docs/en/upgrade/elasticsearch-upgrade.mdx create mode 100644 docs/en/upgrade/index.mdx diff --git a/docs/en/upgrade/elasticsearch-upgrade.mdx b/docs/en/upgrade/elasticsearch-upgrade.mdx new file mode 100644 index 0000000..27c0837 --- /dev/null +++ b/docs/en/upgrade/elasticsearch-upgrade.mdx @@ -0,0 +1,397 @@ +--- +weight: 10 +--- + +# Elasticsearch Upgrade Guide + +## Introduction + +This guide upgrades a cluster that stores logs with **Alauda Container Platform Log Storage for Elasticsearch** to ACP 4.4, where new log data is written to ClickHouse or to an OpenSearch 3.7.0 cluster. + +Elasticsearch and its volumes stay as they are during the upgrade, and new logs go to the new storage, so log collection is not interrupted: + +1. The platform first creates a new log receiving and storage path that runs alongside the existing one, so collectors keep running. +2. Once the new one is ready, new log, event, and audit data are received and written by it. +3. Data that has already queued up is consumed completely. The Elasticsearch cluster keeps running during this time. +4. After you confirm that new data can be queried, uninstall the **Alauda Container Platform Log Storage for Elasticsearch** plugin. Its volumes are retained. +5. If you need the historical data, migrate the retained data to the new storage, and decide whether to delete it after validation. + +:::warning +Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs and PVs before this guide tells you to. Doing it earlier can make the historical data unavailable, or prevent the source metadata snapshot from being captured. +::: + +## Scenarios + +Use this guide when the cluster matches the source and target below. + +| Item | Value | +| --- | --- | +| Source | A workload cluster with **Alauda Container Platform Log Storage for Elasticsearch** installed and running | +| Target platform | ACP 4.4 | +| Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | +| Supported source ACP versions | 4.1.x, 4.2.x, 4.3.x | + +Do not use this guide for a fresh installation, or when the cluster already stores logs in ClickHouse or OpenSearch. See [Installation](../install_log.mdx) instead. + +## Prerequisites + +Before you start, ensure that: + +1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. +2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. +3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: + - An OpenSearch 3.7.0 cluster, or a ClickHouse cluster, sized for both the new traffic and the data that you migrate. + - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. + - For how to install and configure the target storage and Kafka, see the external storage and Kafka installation guide (link to be added). +4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. + +## Upgrade Procedure + +### Step 1: Prepare the target storage, Kafka, and connection Secrets + +Create two Secrets in `cpaas-system` on the workload cluster: one for the target storage, and one for the new Kafka service. Do not overwrite or reuse the Secrets of the legacy Elasticsearch and Kafka. + +#### Target OpenSearch + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoints: "https://:9200" # OpenSearch addresses, separated by commas if there are several; put the highly available coordinator or load balancer address first + username: "" # Can be omitted when the target allows anonymous access + password: "" # Can be omitted when the target allows anonymous access + tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the worker can verify the target before writing + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +--- +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + bootstrap: "" # Kafka addresses in host:port form, separated by commas + kafkaClusterName: "" # Kafka broker resource name; must match the actual name + username: "" # Kafka user name + password: "" # At least 32 characters on Alauda OS nodes or other FIPS-enabled hosts + sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 + topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC + topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC + topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC + tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +`endpoints` accepts several addresses. The write side uses all of them in turn, while the query side uses only the first, so put the highly available coordinator or load balancer address first. + +#### Target ClickHouse + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoint: "https://:8443" # ClickHouse address, including protocol and port + cluster: "" # ClickHouse cluster name + database: "observability" # Target database name, defaults to observability + username: "" # ClickHouse user name + password: "" # ClickHouse password + tls.ca: |- # Required when the target uses HTTPS with a private CA + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +The new Kafka service uses the same `platform-default-mq-conn` as in the OpenSearch section. + +When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` in the target storage connection Secret: the migration tool refuses to start and requires a verifiable `tls.ca` instead. + +### Step 2: Create the new data path + +Create one `PlatformLogForward` for your target. It must use `installMode: Fresh` and the `log.alauda.io/legacy-es-upgrade: "true"` annotation. Do not use `installMode: Adopt`. + +#### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change + namespace: cpaas-system + annotations: + log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + externalStorage: + type: opensearch # Target storage type + secretRef: + name: platform-default-os-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +#### Target ClickHouse + +The `output.type` field is required for this target. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change + namespace: cpaas-system + annotations: + log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + output: + type: clickhouse # Required when the target is ClickHouse + externalStorage: + type: clickhouse # Target storage type + shards: 1 # Actual shard count of the target ClickHouse + replicas: 1 # Actual replica count of the target ClickHouse + secretRef: + name: platform-default-ch-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. + +Save the YAML as `platform-log-forward.yaml` and apply it: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +The new data path is not available immediately. The platform first creates it, then switches the log entry point, and finally waits for the data that queued up in the old cluster to be consumed; the time this takes depends on the backlog. Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl -n cpaas-system get platformlogforward platform-default -w +``` + +The `Phase` column changes from `Provisioning` to `Ready`, and the `Ready` column becomes `True` at the same time. Continue only after you see `Ready`. + +To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl -n cpaas-system get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeCompleted`, the log entry point has switched to the new data path and the data queued in the old cluster has been consumed, so this step is complete. If the reason is `Blocked`, the `message` explains why. + +Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. + +### Step 3: Uninstall the Elasticsearch storage plugin + +:::warning +Uninstall the plugin only when the new data path is `Ready` and the `LegacyESUpgrade` condition is complete. Use the controlled uninstall below: do not delete the plugin from the console, do not force-delete resources, and do not remove finalizers. +::: + +The script below performs the uninstall serially. Run it in `bash` as a single copy-paste block instead of entering the commands one by one: the platform rewrites the plugin discovery flag automatically, so typing them one at a time may not leave enough time to delete the plugin instance before it is restored. Set `CLUSTER` to the workload cluster name as registered in the `global` cluster. + +```bash +set -euo pipefail + +export GLOBAL_KUBECONFIG= +export WORKLOAD_KUBECONFIG= +CLUSTER= + +g() { kubectl --kubeconfig "$GLOBAL_KUBECONFIG" "$@"; } +w() { kubectl --kubeconfig "$WORKLOAD_KUBECONFIG" "$@"; } + +# Resolve the plugin instance name and the plugin config name for this cluster +MODULE_INFO="$(g get moduleplugin logcenter \ + -o jsonpath="{.status.installed[?(@.cluster==\"$CLUSTER\")].name}")" +MODULE_CONFIG="$(g get moduleplugin logcenter \ + -o jsonpath='{.status.moduleConfigs[0].name}')" + +if [ -n "$MODULE_INFO" ]; then + # Turn off the discovery flags + g patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + g patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + + # Delete the plugin instance and wait until it is gone + g delete moduleinfo "$MODULE_INFO" --ignore-not-found + g wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No plugin instance found for cluster $CLUSTER; it may already be uninstalled. Continuing to verify that the plugin instance is gone." +fi + +# Delete the per-cluster plugin install record, then wait for the plugin instance on the workload cluster to disappear +w delete clusterplugininstance logcenter --ignore-not-found +w -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +`ClusterPluginInstance/logcenter` must be deleted. If it remains, the platform recreates `ModuleInfo`. If the script fails halfway, the platform writes `labelCluster` back to `true` automatically, so no half-finished state is left behind; fix the cause and run it again. + +### Step 4: Migrate historical data + +Skip this step only when the upgrade plan clearly states that the Elasticsearch historical data is not required. Decide this before you uninstall the plugin in Step 3, and keep the source volumes until the migration completes or you confirm that you discard the data. + +Create one `LegacyESMigration` for your target. + +#### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Migration resource name, used by the commands below + namespace: cpaas-system +spec: + image: # Required, migration worker image, including registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate, cannot be changed after creation + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + - "audit-20260825" + target: + type: opensearch # Target storage type, keep it consistent with PlatformLogForward, cannot be changed after creation + secretRef: + name: platform-default-os-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 + syncIntervalSeconds: 5 # Minimum wait between batches, in seconds + maxConcurrentJobs: 1 # Concurrent jobs, up to 2 for OpenSearch +``` + +#### Target ClickHouse + +Change `target.type` to `clickhouse`, point the Secret at `platform-default-ch-conn`, and keep `maxConcurrentJobs` at `1`. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Migration resource name, used by the commands below + namespace: cpaas-system +spec: + image: # Required, migration worker image, including registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate, cannot be changed after creation + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + - "audit-20260825" + target: + type: clickhouse # Target storage type, keep it consistent with PlatformLogForward, cannot be changed after creation + secretRef: + name: platform-default-ch-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 + syncIntervalSeconds: 5 # Minimum wait between batches, in seconds + maxConcurrentJobs: 1 # Concurrent jobs, 1 for ClickHouse only +``` + +Each `indexScope` entry corresponds to one data category. Use a wildcard to migrate a whole category, or name a single day. Index names use the format `-`. + +| `indexScope` value | Console item | Data | Full index name example | +| --- | --- | --- | --- | +| `log-workload-*` | Log Workload | Application and container logs | `log-workload-20260825` | +| `log-platform-*` | Log Platform | Platform component logs | `log-platform-20260825` | +| `log-system-*` | Log System | System logs | `log-system-20260825` | +| `log-kubernetes-*` | Log Kubernetes | Kubernetes logs | `log-kubernetes-20260825` | +| `event-*` | Kubernetes Event | Kubernetes events | `event-20260825` | +| `audit-*` | Audit | Audit logs | `audit-20260825` | + +The examples above cover these six categories. If the source also contains per-project log indices or metering data, add them to `indexScope` as needed, otherwise that data is not migrated: + +- `log-project-*`, workload logs split by project, for example `log-project--20260825`. Some sources contain both `log-workload-*` and these indices. +- `meter-*`, metering data, for example `meter-20260825`. + +Save the YAML as `legacy-es-migration.yaml`, apply it, and wait for the migration to complete: + +```bash +kubectl -n cpaas-system apply -f legacy-es-migration.yaml + +# Watch the phase, press Ctrl+C to stop +kubectl -n cpaas-system get legacyesmigration platform-es-history -w + +# Check the progress and the task status of each volume +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' +kubectl -n cpaas-system get jobs,pods -l log.alauda.io/legacy-es-migration +``` + +**Verification:** `status.phase` is `Succeeded` and the task of every volume succeeded. + +If the phase stays at `Blocked`, use the commands below to see why, and fix the reported source volume, node, target connection, or source data issue: + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' +``` + +Do not delete the migration resource or its source volumes. + +## Remove the retained Elasticsearch volumes + +:::warning +Deleting the retained PVCs and PVs permanently removes the source of the historical migration. Do it only after `LegacyESMigration.status.phase` is `Succeeded` and the target queries pass, or after an approved decision to discard the historical data has been recorded. +::: + +First export the records for your files and list the volumes retained by this upgrade: + +```bash +mkdir -p ./es-upgrade-cleanup + +kubectl -n cpaas-system get pvc \ + -l 'service_name in (cpaas-elasticsearch,cpaas-elasticsearch-master)' \ + -o yaml > ./es-upgrade-cleanup/legacy-es-pvcs.yaml +kubectl get pv -l log.alauda.io/legacy-es-data-protected=true -o yaml \ + > ./es-upgrade-cleanup/legacy-es-pvs.yaml + +kubectl -n cpaas-system get pvc -l 'service_name in (cpaas-elasticsearch,cpaas-elasticsearch-master)' +kubectl get pv -l log.alauda.io/legacy-es-data-protected=true +``` + +After you confirm that the volumes in the list belong to this upgrade, approve and delete them one at a time: + +```bash +LEGACY_ES_PVC='' +LEGACY_ES_PV='' + +kubectl -n cpaas-system annotate pvc "$LEGACY_ES_PVC" \ + log.alauda.io/legacy-es-data-delete-approved=true --overwrite +kubectl annotate pv "$LEGACY_ES_PV" \ + log.alauda.io/legacy-es-data-delete-approved=true --overwrite + +kubectl -n cpaas-system delete pvc "$LEGACY_ES_PVC" +kubectl delete pv "$LEGACY_ES_PV" +``` + +Repeat the commands only for the volumes that belong to this upgrade. If a PVC is stuck in `Terminating`, do not delete the `kubernetes.io/pvc-protection` or `kubernetes.io/pv-protection` finalizer; check the approval annotation and the validating webhook first. + +Local host-path data is not deleted automatically. Handle it separately through your storage cleanup procedure after the export records are complete. diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx new file mode 100644 index 0000000..23f8469 --- /dev/null +++ b/docs/en/upgrade/index.mdx @@ -0,0 +1,19 @@ +--- +weight: 15 +--- + +# Upgrade + +This section explains how to upgrade the Logging components of an existing ACP deployment. + +Find your current deployment in the table below, then follow the matching guide. + +| Current deployment | Upgrade to | Guide | +| --- | --- | --- | +| Logs are stored with **Alauda Container Platform Log Storage for Elasticsearch** | ACP 4.4 with ClickHouse or an OpenSearch 3.7.0 cluster | [Elasticsearch Upgrade Guide](./elasticsearch-upgrade.mdx) | + + + +:::info +If the cluster already stores logs in ClickHouse or OpenSearch, no upgrade guide is needed. Follow [Installation](../install_log.mdx) to install or update the plugins. +::: From 33947b7b015c1c871a6b8a59218777b2a2142e43 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 13:36:43 +0800 Subject: [PATCH 02/38] docs: refine Elasticsearch upgrade uninstall steps --- docs/en/upgrade/elasticsearch-upgrade.mdx | 86 +++++++++++++++++------ 1 file changed, 66 insertions(+), 20 deletions(-) diff --git a/docs/en/upgrade/elasticsearch-upgrade.mdx b/docs/en/upgrade/elasticsearch-upgrade.mdx index 27c0837..b6adbfc 100644 --- a/docs/en/upgrade/elasticsearch-upgrade.mdx +++ b/docs/en/upgrade/elasticsearch-upgrade.mdx @@ -13,7 +13,7 @@ Elasticsearch and its volumes stay as they are during the upgrade, and new logs 1. The platform first creates a new log receiving and storage path that runs alongside the existing one, so collectors keep running. 2. Once the new one is ready, new log, event, and audit data are received and written by it. 3. Data that has already queued up is consumed completely. The Elasticsearch cluster keeps running during this time. -4. After you confirm that new data can be queried, uninstall the **Alauda Container Platform Log Storage for Elasticsearch** plugin. Its volumes are retained. +4. After you confirm that new data can be queried, confirm that the upgrade program has protected the legacy Elasticsearch PVCs and PVs, then uninstall the **Alauda Container Platform Log Storage for Elasticsearch** plugin. Its protected volumes are retained. 5. If you need the historical data, migrate the retained data to the new storage, and decide whether to delete it after validation. :::warning @@ -208,42 +208,88 @@ Once this step is complete, produce or locate new log, event, and audit records, Uninstall the plugin only when the new data path is `Ready` and the `LegacyESUpgrade` condition is complete. Use the controlled uninstall below: do not delete the plugin from the console, do not force-delete resources, and do not remove finalizers. ::: -The script below performs the uninstall serially. Run it in `bash` as a single copy-paste block instead of entering the commands one by one: the platform rewrites the plugin discovery flag automatically, so typing them one at a time may not leave enough time to delete the plugin instance before it is restored. Set `CLUSTER` to the workload cluster name as registered in the `global` cluster. +:::warning +Before you continue, confirm that the upgrade program has protected the legacy Elasticsearch PVCs and their bound PVs. If the PV reclaim policy is not `Retain`, or if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, stop and contact support. Uninstalling the plugin can delete an unprotected volume. Do not add or modify this protection manually. +::: + +The controlled uninstall is a short sequence: clear the plugin discovery flags, delete the target `ModuleInfo`, then delete `ClusterPluginInstance/logcenter`. The console uninstall also deletes `ModuleInfo`, but this upgrade needs the controlled sequence below so that the discovery flags and the per-cluster install record are handled in the correct order. + +There is one timing requirement. The risk is not the `labelCluster` rewrite itself: base-operator may restore that field, but it does not recreate `ModuleInfo`. The real risk is `ClusterPluginInstance/logcenter`; cluster-transformer can use it to recreate `ModuleInfo` on a later reconciliation. The window starts when `ModuleInfo` is gone while the install record still exists, so the `ModuleInfo` delete and the `ClusterPluginInstance/logcenter` delete must run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. Once the install record is gone, a `labelCluster` rewrite is harmless. + +**On the global cluster: start the critical sequence** + +Set `CLUSTER` to the workload cluster name as registered in the global cluster, then run these commands: ```bash set -euo pipefail -export GLOBAL_KUBECONFIG= -export WORKLOAD_KUBECONFIG= CLUSTER= -g() { kubectl --kubeconfig "$GLOBAL_KUBECONFIG" "$@"; } -w() { kubectl --kubeconfig "$WORKLOAD_KUBECONFIG" "$@"; } - -# Resolve the plugin instance name and the plugin config name for this cluster -MODULE_INFO="$(g get moduleplugin logcenter \ +# 1. Resolve the plugin instance name and the plugin config name for this cluster +MODULE_INFO="$(kubectl get moduleplugin logcenter \ -o jsonpath="{.status.installed[?(@.cluster==\"$CLUSTER\")].name}")" -MODULE_CONFIG="$(g get moduleplugin logcenter \ +MODULE_CONFIG="$(kubectl get moduleplugin logcenter \ -o jsonpath='{.status.moduleConfigs[0].name}')" if [ -n "$MODULE_INFO" ]; then - # Turn off the discovery flags - g patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' - g patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + # 2. Turn off the discovery flags + kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' - # Delete the plugin instance and wait until it is gone - g delete moduleinfo "$MODULE_INFO" --ignore-not-found - g wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m + # 3. Return as soon as the delete request is accepted; do not wait here + kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false else echo "No plugin instance found for cluster $CLUSTER; it may already be uninstalled. Continuing to verify that the plugin instance is gone." fi +``` + +**On the workload cluster: finish the critical sequence** -# Delete the per-cluster plugin install record, then wait for the plugin instance on the workload cluster to disappear -w delete clusterplugininstance logcenter --ignore-not-found -w -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +Switch the current kubectl context to the workload cluster immediately. Do not run any wait or verification first. Run: + +```bash +# 4. Delete the per-cluster install record before it can recreate ModuleInfo +kubectl delete clusterplugininstance logcenter --ignore-not-found +``` + +**Wait for the old data path to be removed** + +Switch back to the global cluster and wait for `ModuleInfo` to disappear: + +```bash +CLUSTER= +MODULE_INFO="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{.status.installed[?(@.cluster==\"$CLUSTER\")].name}")" + +if [ -n "$MODULE_INFO" ]; then + kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No ModuleInfo remains for cluster $CLUSTER." +fi +``` + +Switch to the workload cluster and wait for the legacy `AppRelease` to be removed: + +```bash +kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +After these steps, wait 60 seconds and verify that the resources stayed gone. Each command must return no resource. + +**On the global cluster** + +```bash +kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found +``` + +**On the workload cluster** + +```bash +kubectl get clusterplugininstance logcenter --ignore-not-found +kubectl -n cpaas-system get apprelease logcenter --ignore-not-found ``` -`ClusterPluginInstance/logcenter` must be deleted. If it remains, the platform recreates `ModuleInfo`. If the script fails halfway, the platform writes `labelCluster` back to `true` automatically, so no half-finished state is left behind; fix the cause and run it again. +If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; that is normal once the per-cluster install record is gone. If the critical sequence fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically; fix the cause and run the sequence again. ### Step 4: Migrate historical data From 9c6e5048be68e9ae4559d1d2dbe8014369fc65f5 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 13:53:30 +0800 Subject: [PATCH 03/38] docs: align ES upgrade with legacy migration gates --- docs/en/upgrade/elasticsearch-upgrade.mdx | 342 ++++++++++++---------- 1 file changed, 183 insertions(+), 159 deletions(-) diff --git a/docs/en/upgrade/elasticsearch-upgrade.mdx b/docs/en/upgrade/elasticsearch-upgrade.mdx index b6adbfc..6ffcc1a 100644 --- a/docs/en/upgrade/elasticsearch-upgrade.mdx +++ b/docs/en/upgrade/elasticsearch-upgrade.mdx @@ -8,13 +8,14 @@ weight: 10 This guide upgrades a cluster that stores logs with **Alauda Container Platform Log Storage for Elasticsearch** to ACP 4.4, where new log data is written to ClickHouse or to an OpenSearch 3.7.0 cluster. -Elasticsearch and its volumes stay as they are during the upgrade, and new logs go to the new storage, so log collection is not interrupted: +Elasticsearch and its volumes stay in place while the platform creates the new path alongside the legacy one, so log collection is not interrupted: -1. The platform first creates a new log receiving and storage path that runs alongside the existing one, so collectors keep running. -2. Once the new one is ready, new log, event, and audit data are received and written by it. -3. Data that has already queued up is consumed completely. The Elasticsearch cluster keeps running during this time. -4. After you confirm that new data can be queried, confirm that the upgrade program has protected the legacy Elasticsearch PVCs and PVs, then uninstall the **Alauda Container Platform Log Storage for Elasticsearch** plugin. Its protected volumes are retained. -5. If you need the historical data, migrate the retained data to the new storage, and decide whether to delete it after validation. +1. The platform creates the new log receiving and storage path and starts writing new log, event, and audit data to it. +2. The legacy Elasticsearch cluster and the old pipeline stay running while queued data is consumed. +3. The PlatformLogForward reports `LegacyESUpgradeCompleted` after cutover and drain complete. +4. If you need the historical data, create the migration resource and wait for `PrecaptureReady` before uninstalling the legacy plugin. +5. Confirm that the upgrade program has protected the legacy Elasticsearch volumes, then uninstall the plugin. +6. Observe the same migration resource until it succeeds, and keep the protected source volumes until validation is complete. :::warning Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs and PVs before this guide tells you to. Doing it earlier can make the historical data unavailable, or prevent the source metadata snapshot from being captured. @@ -42,8 +43,24 @@ Before you start, ensure that: 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - An OpenSearch 3.7.0 cluster, or a ClickHouse cluster, sized for both the new traffic and the data that you migrate. - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - - For how to install and configure the target storage and Kafka, see the external storage and Kafka installation guide (link to be added). + - The target storage and Kafka connection details prepared according to their product documentation. 4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. +5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. +6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. + + +## Recommended Sequence + +Use this order. Each gate must pass before the next step starts. + +| Step | Where | Action | Gate to continue | +| --- | --- | --- | --- | +| 1 | Workload cluster | Prepare the target storage, Kafka, and connection Secrets | The target storage and topics are reachable | +| 2 | Workload cluster | Create `PlatformLogForward` | `Phase=Ready` and `LegacyESUpgradeCompleted` | +| 3 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | +| 4 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | Those three objects stay absent for 60 seconds and new log queries still pass | +| 5 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | +| 6 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | ## Upgrade Procedure @@ -128,7 +145,6 @@ apiVersion: log.alauda.io/v1alpha1 kind: PlatformLogForward metadata: name: platform-default # Fixed cluster singleton name, do not change - namespace: cpaas-system annotations: log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow spec: @@ -154,7 +170,6 @@ apiVersion: log.alauda.io/v1alpha1 kind: PlatformLogForward metadata: name: platform-default # Fixed cluster singleton name, do not change - namespace: cpaas-system annotations: log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow spec: @@ -177,6 +192,8 @@ spec: `externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. + Save the YAML as `platform-log-forward.yaml` and apply it: ```bash @@ -186,7 +203,7 @@ kubectl apply -f platform-log-forward.yaml The new data path is not available immediately. The platform first creates it, then switches the log entry point, and finally waits for the data that queued up in the old cluster to be consumed; the time this takes depends on the backlog. Watch the status until it finishes, and press `Ctrl+C` to stop: ```bash -kubectl -n cpaas-system get platformlogforward platform-default -w +kubectl get platformlogforward platform-default -w ``` The `Phase` column changes from `Provisioning` to `Ready`, and the `Ready` column becomes `True` at the same time. Continue only after you see `Ready`. @@ -194,7 +211,7 @@ The `Phase` column changes from `Provisioning` to `Ready`, and the `Ready` colum To follow the progress or troubleshoot, read the status conditions: ```bash -kubectl -n cpaas-system get platformlogforward platform-default \ +kubectl get platformlogforward platform-default \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` @@ -202,19 +219,124 @@ Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeComple Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. -### Step 3: Uninstall the Elasticsearch storage plugin +If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform consumes the queued messages through the old path and records `LegacyKafkaDrained` when the old consumer lag reaches zero. `LegacyESUpgradeCompleted` is the gate for this procedure. + +### Step 3: Prepare the historical data migration (optional) + +Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. + +If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. This lets the platform select the final precapture and resolve all source PVCs while the legacy ES StatefulSet still exists. Creating the migration only after the uninstall requires explicit source PVC references and cannot use the final precapture. + +#### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required, approved worker image with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # Must match the PlatformLogForward target + secretRef: + name: platform-default-os-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 + syncIntervalSeconds: 5 # Minimum wait between batches, in seconds + maxConcurrentJobs: 1 # Concurrent jobs, up to 2 for OpenSearch +``` + +#### Target ClickHouse + +Change `target.type` to `clickhouse`, point the Secret at `platform-default-ch-conn`, and keep `maxConcurrentJobs` at `1`. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required, approved worker image with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # Must match the PlatformLogForward target + secretRef: + name: platform-default-ch-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 + syncIntervalSeconds: 5 # Minimum wait between batches, in seconds + maxConcurrentJobs: 1 # Concurrent jobs, 1 for ClickHouse only +``` + +The standard `indexScope` categories are: + +| `indexScope` value | Data | Full index name example | +| --- | --- | --- | +| `log-workload-*` | Application and container logs | `log-workload-20260825` | +| `log-platform-*` | Platform component logs | `log-platform-20260825` | +| `log-system-*` | System logs | `log-system-20260825` | +| `log-kubernetes-*` | Kubernetes logs | `log-kubernetes-20260825` | +| `event-*` | Kubernetes events | `event-20260825` | +| `audit-*` | Audit logs | `audit-20260825` | + +If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections are immutable after creation. + +In this recommended flow, do not set `source.pvcRefs`. While the legacy StatefulSet still exists, the platform discovers every source PVC and records the result in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. Creating the migration after the uninstall requires explicit source PVC references and cannot use the final precapture. + +Apply the resource and wait for `PrecaptureReady`: + +```bash +kubectl -n cpaas-system apply -f legacy-es-migration.yaml + +# Watch the phase; press Ctrl+C to stop +kubectl -n cpaas-system get legacyesmigration platform-es-history -w + +# Show the conditions if the phase does not advance +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. + +Do not delete the migration resource or its source volumes. + +### Step 4: Uninstall the legacy Elasticsearch storage plugin :::warning -Uninstall the plugin only when the new data path is `Ready` and the `LegacyESUpgrade` condition is complete. Use the controlled uninstall below: do not delete the plugin from the console, do not force-delete resources, and do not remove finalizers. +Run this step only when all of these conditions are true: + +- `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. +- If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. +- The upgrade program has confirmed that the legacy Elasticsearch PVC and PV protection is in place. If the PV reclaim policy is not `Retain`, or if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, stop and contact support. Do not add or modify this protection manually. +- If the source uses legacy Kafka, the platform has recorded that the old consumer lag is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. ::: :::warning -Before you continue, confirm that the upgrade program has protected the legacy Elasticsearch PVCs and their bound PVs. If the PV reclaim policy is not `Retain`, or if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, stop and contact support. Uninstalling the plugin can delete an unprotected volume. Do not add or modify this protection manually. +The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` and its `ModuleConfig` before deleting the target `ModuleInfo`. Clearing the field is required because the existing logagent still records a cross-cluster dependency on `logcenter`; it bypasses that dependency check so the target `ModuleInfo` can be removed. Do not modify or delete logagent or its `crossClusterDependency`. The temporary window affects logcenter cluster-label discovery, so run this in the approved change window. ::: -The controlled uninstall is a short sequence: clear the plugin discovery flags, delete the target `ModuleInfo`, then delete `ClusterPluginInstance/logcenter`. The console uninstall also deletes `ModuleInfo`, but this upgrade needs the controlled sequence below so that the discovery flags and the per-cluster install record are handled in the correct order. +If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected with `moduleinfo is depended by ...`, or the platform team asks you to run the controlled procedure, use the sequence below. The controlled sequence clears the discovery fields and the per-cluster install record in the order required for this upgrade. -There is one timing requirement. The risk is not the `labelCluster` rewrite itself: base-operator may restore that field, but it does not recreate `ModuleInfo`. The real risk is `ClusterPluginInstance/logcenter`; cluster-transformer can use it to recreate `ModuleInfo` on a later reconciliation. The window starts when `ModuleInfo` is gone while the install record still exists, so the `ModuleInfo` delete and the `ClusterPluginInstance/logcenter` delete must run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. Once the install record is gone, a `labelCluster` rewrite is harmless. +The critical part is a short sequence. The durable problem is not the `labelCluster` rewrite itself; base-operator may restore that field, but it does not recreate `ModuleInfo`. The real risk is a surviving `ClusterPluginInstance/logcenter`, because cluster-transformer can use it to recreate `ModuleInfo` on a later reconciliation. The `ModuleInfo` delete and `ClusterPluginInstance/logcenter` delete must therefore run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. **On the global cluster: start the critical sequence** @@ -225,21 +347,33 @@ set -euo pipefail CLUSTER= -# 1. Resolve the plugin instance name and the plugin config name for this cluster -MODULE_INFO="$(kubectl get moduleplugin logcenter \ - -o jsonpath="{.status.installed[?(@.cluster==\"$CLUSTER\")].name}")" -MODULE_CONFIG="$(kubectl get moduleplugin logcenter \ - -o jsonpath='{.status.moduleConfigs[0].name}')" +# 1. Resolve the target ModuleInfo. Stop if the result is ambiguous. +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" +if [ "$MODULE_COUNT" -gt 1 ]; then + echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 + exit 1 +fi -if [ -n "$MODULE_INFO" ]; then - # 2. Turn off the discovery flags +if [ "$MODULE_COUNT" -eq 1 ]; then + MODULE_INFO="$MODULE_INFOS" + MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" + MODULE_CONFIG="logcenter-${MODULE_VERSION}" + + # 2. Record the current management objects before changing them. + kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml + kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml + kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml + + # 3. Clear the discovery flags. This is also what bypasses the logagent dependency check. kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' - # 3. Return as soon as the delete request is accepted; do not wait here + # 4. Return as soon as the delete request is accepted. Do not wait here. kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false else - echo "No plugin instance found for cluster $CLUSTER; it may already be uninstalled. Continuing to verify that the plugin instance is gone." + echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." fi ``` @@ -248,7 +382,7 @@ fi Switch the current kubectl context to the workload cluster immediately. Do not run any wait or verification first. Run: ```bash -# 4. Delete the per-cluster install record before it can recreate ModuleInfo +# 5. Delete the per-cluster install record before it can recreate ModuleInfo. kubectl delete clusterplugininstance logcenter --ignore-not-found ``` @@ -258,8 +392,9 @@ Switch back to the global cluster and wait for `ModuleInfo` to disappear: ```bash CLUSTER= -MODULE_INFO="$(kubectl get moduleplugin logcenter \ - -o jsonpath="{.status.installed[?(@.cluster==\"$CLUSTER\")].name}")" +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" if [ -n "$MODULE_INFO" ]; then kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m @@ -289,155 +424,44 @@ kubectl get clusterplugininstance logcenter --ignore-not-found kubectl -n cpaas-system get apprelease logcenter --ignore-not-found ``` -If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; that is normal once the per-cluster install record is gone. If the critical sequence fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically; fix the cause and run the sequence again. - -### Step 4: Migrate historical data - -Skip this step only when the upgrade plan clearly states that the Elasticsearch historical data is not required. Decide this before you uninstall the plugin in Step 3, and keep the source volumes until the migration completes or you confirm that you discard the data. - -Create one `LegacyESMigration` for your target. - -#### Target OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # Migration resource name, used by the commands below - namespace: cpaas-system -spec: - image: # Required, migration worker image, including registry and tag or digest - source: - indexScope: # Required, selects the historical indices to migrate, cannot be changed after creation - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - - "audit-20260825" - target: - type: opensearch # Target storage type, keep it consistent with PlatformLogForward, cannot be changed after creation - secretRef: - name: platform-default-os-conn # The same connection Secret as PlatformLogForward - namespace: cpaas-system - options: - batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 - syncIntervalSeconds: 5 # Minimum wait between batches, in seconds - maxConcurrentJobs: 1 # Concurrent jobs, up to 2 for OpenSearch -``` - -#### Target ClickHouse - -Change `target.type` to `clickhouse`, point the Secret at `platform-default-ch-conn`, and keep `maxConcurrentJobs` at `1`. - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # Migration resource name, used by the commands below - namespace: cpaas-system -spec: - image: # Required, migration worker image, including registry and tag or digest - source: - indexScope: # Required, selects the historical indices to migrate, cannot be changed after creation - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - - "audit-20260825" - target: - type: clickhouse # Target storage type, keep it consistent with PlatformLogForward, cannot be changed after creation - secretRef: - name: platform-default-ch-conn # The same connection Secret as PlatformLogForward - namespace: cpaas-system - options: - batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 - syncIntervalSeconds: 5 # Minimum wait between batches, in seconds - maxConcurrentJobs: 1 # Concurrent jobs, 1 for ClickHouse only -``` - -Each `indexScope` entry corresponds to one data category. Use a wildcard to migrate a whole category, or name a single day. Index names use the format `-`. - -| `indexScope` value | Console item | Data | Full index name example | -| --- | --- | --- | --- | -| `log-workload-*` | Log Workload | Application and container logs | `log-workload-20260825` | -| `log-platform-*` | Log Platform | Platform component logs | `log-platform-20260825` | -| `log-system-*` | Log System | System logs | `log-system-20260825` | -| `log-kubernetes-*` | Log Kubernetes | Kubernetes logs | `log-kubernetes-20260825` | -| `event-*` | Kubernetes Event | Kubernetes events | `event-20260825` | -| `audit-*` | Audit | Audit logs | `audit-20260825` | +If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; do not manually restore them, and verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically. Do not run the workload-cluster block for that failed attempt; fix the cause and run the global sequence again. -The examples above cover these six categories. If the source also contains per-project log indices or metering data, add them to `indexScope` as needed, otherwise that data is not migrated: +After the stability check passes, produce a new log, event, and audit record and confirm that it is queryable from the new target. Confirm that the platform's Razor callback and log query paths still work. Do not continue to migration verification if the new data path is unhealthy. -- `log-project-*`, workload logs split by project, for example `log-project--20260825`. Some sources contain both `log-workload-*` and these indices. -- `meter-*`, metering data, for example `meter-20260825`. +### Step 5: Complete and verify the historical migration (optional) -Save the YAML as `legacy-es-migration.yaml`, apply it, and wait for the migration to complete: +If you created `LegacyESMigration`, continue observing the same resource. Do not delete or recreate it. The phase should move from `PrecaptureReady` to `Validating`, `Running` (or `Retrying`), and finally `Succeeded`. ```bash -kubectl -n cpaas-system apply -f legacy-es-migration.yaml - -# Watch the phase, press Ctrl+C to stop kubectl -n cpaas-system get legacyesmigration platform-es-history -w -# Check the progress and the task status of each volume kubectl -n cpaas-system get legacyesmigration platform-es-history \ -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' -kubectl -n cpaas-system get jobs,pods -l log.alauda.io/legacy-es-migration -``` - -**Verification:** `status.phase` is `Succeeded` and the task of every volume succeeded. - -If the phase stays at `Blocked`, use the commands below to see why, and fix the reported source volume, node, target connection, or source data issue: - -```bash -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' kubectl -n cpaas-system get legacyesmigration platform-es-history \ -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' ``` -Do not delete the migration resource or its source volumes. - -## Remove the retained Elasticsearch volumes - -:::warning -Deleting the retained PVCs and PVs permanently removes the source of the historical migration. Do it only after `LegacyESMigration.status.phase` is `Succeeded` and the target queries pass, or after an approved decision to discard the historical data has been recorded. -::: - -First export the records for your files and list the volumes retained by this upgrade: +**Verification:** -```bash -mkdir -p ./es-upgrade-cleanup +- `status.phase` is `Succeeded`. +- Every entry in `status.tasks` succeeded. +- Sample queries against the new target return the expected history for each `indexScope` category. +- The protected source PVCs and PVs are still present and bound. -kubectl -n cpaas-system get pvc \ - -l 'service_name in (cpaas-elasticsearch,cpaas-elasticsearch-master)' \ - -o yaml > ./es-upgrade-cleanup/legacy-es-pvcs.yaml -kubectl get pv -l log.alauda.io/legacy-es-data-protected=true -o yaml \ - > ./es-upgrade-cleanup/legacy-es-pvs.yaml +If the phase is `Blocked` or `Failed`, do not delete the migration resource, Jobs, or source volumes; read the conditions and task status, then contact support. -kubectl -n cpaas-system get pvc -l 'service_name in (cpaas-elasticsearch,cpaas-elasticsearch-master)' -kubectl get pv -l log.alauda.io/legacy-es-data-protected=true -``` +If no migration was created, record the explicit decision that historical data is not required, and keep the protected source volumes until that decision is approved. -After you confirm that the volumes in the list belong to this upgrade, approve and delete them one at a time: +### Step 6: Retained volumes -```bash -LEGACY_ES_PVC='' -LEGACY_ES_PV='' +The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until migration and target validation are complete. Do not remove protection annotations, delete PVCs or PVs, or remove finalizers. To release the volumes after validation, contact Alauda support or follow the separate approved cleanup procedure. -kubectl -n cpaas-system annotate pvc "$LEGACY_ES_PVC" \ - log.alauda.io/legacy-es-data-delete-approved=true --overwrite -kubectl annotate pv "$LEGACY_ES_PV" \ - log.alauda.io/legacy-es-data-delete-approved=true --overwrite - -kubectl -n cpaas-system delete pvc "$LEGACY_ES_PVC" -kubectl delete pv "$LEGACY_ES_PV" -``` +Only the ES PVCs and PVs are retained by this upgrade. Legacy Kafka and ZooKeeper volumes are not part of the retention scope. -Repeat the commands only for the volumes that belong to this upgrade. If a PVC is stuck in `Terminating`, do not delete the `kubernetes.io/pvc-protection` or `kubernetes.io/pv-protection` finalizer; check the approval annotation and the validating webhook first. +## If a Step Is Blocked -Local host-path data is not deleted automatically. Handle it separately through your storage cleanup procedure after the export records are complete. +- `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. +- Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. +- `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. +- `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. +- New log queries fail after uninstall: stop the migration and contact support. Do not delete the retained volumes. From 373fece8aff13cb21db99b827bdcb014b5e4a046 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 14:18:43 +0800 Subject: [PATCH 04/38] docs: make the ES upgrade guide implementation-ready --- docs/en/upgrade/elasticsearch-upgrade.mdx | 41 +++++++++++++++++++++-- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/docs/en/upgrade/elasticsearch-upgrade.mdx b/docs/en/upgrade/elasticsearch-upgrade.mdx index 6ffcc1a..65d2679 100644 --- a/docs/en/upgrade/elasticsearch-upgrade.mdx +++ b/docs/en/upgrade/elasticsearch-upgrade.mdx @@ -34,6 +34,15 @@ Use this guide when the cluster matches the source and target below. Do not use this guide for a fresh installation, or when the cluster already stores logs in ClickHouse or OpenSearch. See [Installation](../install_log.mdx) instead. +## Who Runs This Procedure + +This is an implementation runbook for Alauda implementation engineers or platform administrators. It is not an end-user self-service procedure. + +- Use a platform account that can access both the global management cluster and the target workload cluster. +- Use a separate terminal or kubectl context for each cluster where possible. Each command block in this guide states which cluster it runs on. +- Before each block, confirm the current context with `kubectl config current-context`. +- In a managed environment, confirm the change window and escalation path with Alauda support before you start. + ## Prerequisites Before you start, ensure that: @@ -49,7 +58,11 @@ Before you start, ensure that: 6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. -## Recommended Sequence +:::warning +During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Do not delete PVCs, PVs, or protection finalizers. Do not delete or recreate a `LegacyESMigration` resource. These actions can make historical data unavailable or invalidate the migration boundary. +::: + +## Upgrade Flow at a Glance Use this order. Each gate must pass before the next step starts. @@ -58,7 +71,7 @@ Use this order. Each gate must pass before the next step starts. | 1 | Workload cluster | Prepare the target storage, Kafka, and connection Secrets | The target storage and topics are reachable | | 2 | Workload cluster | Create `PlatformLogForward` | `Phase=Ready` and `LegacyESUpgradeCompleted` | | 3 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | -| 4 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | Those three objects stay absent for 60 seconds and new log queries still pass | +| 4 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | | 5 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | | 6 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | @@ -66,6 +79,8 @@ Use this order. Each gate must pass before the next step starts. ### Step 1: Prepare the target storage, Kafka, and connection Secrets +**Run on the workload cluster.** + Create two Secrets in `cpaas-system` on the workload cluster: one for the target storage, and one for the new Kafka service. Do not overwrite or reuse the Secrets of the legacy Elasticsearch and Kafka. #### Target OpenSearch @@ -136,6 +151,8 @@ When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` ### Step 2: Create the new data path +**Run on the workload cluster.** + Create one `PlatformLogForward` for your target. It must use `installMode: Fresh` and the `log.alauda.io/legacy-es-upgrade: "true"` annotation. Do not use `installMode: Adopt`. #### Target OpenSearch @@ -223,6 +240,8 @@ If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeep ### Step 3: Prepare the historical data migration (optional) +**Run on the workload cluster.** + Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. This lets the platform select the final precapture and resolve all source PVCs while the legacy ES StatefulSet still exists. Creating the migration only after the uninstall requires explicit source PVC references and cannot use the final precapture. @@ -336,7 +355,7 @@ The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` a If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected with `moduleinfo is depended by ...`, or the platform team asks you to run the controlled procedure, use the sequence below. The controlled sequence clears the discovery fields and the per-cluster install record in the order required for this upgrade. -The critical part is a short sequence. The durable problem is not the `labelCluster` rewrite itself; base-operator may restore that field, but it does not recreate `ModuleInfo`. The real risk is a surviving `ClusterPluginInstance/logcenter`, because cluster-transformer can use it to recreate `ModuleInfo` on a later reconciliation. The `ModuleInfo` delete and `ClusterPluginInstance/logcenter` delete must therefore run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. +The critical part is a short sequence. The platform may restore the `labelCluster` field after it is cleared; that is not the durable problem. The real risk is a surviving `ClusterPluginInstance/logcenter`, which the platform can use to recreate `ModuleInfo` later. The `ModuleInfo` delete and `ClusterPluginInstance/logcenter` delete must therefore run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. **On the global cluster: start the critical sequence** @@ -430,6 +449,8 @@ After the stability check passes, produce a new log, event, and audit record and ### Step 5: Complete and verify the historical migration (optional) +**Run on the workload cluster.** + If you created `LegacyESMigration`, continue observing the same resource. Do not delete or recreate it. The phase should move from `PrecaptureReady` to `Validating`, `Running` (or `Retrying`), and finally `Succeeded`. ```bash @@ -458,6 +479,20 @@ The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until mi Only the ES PVCs and PVs are retained by this upgrade. Legacy Kafka and ZooKeeper volumes are not part of the retention scope. +Export or record the final PLF conditions, migration status, target query results, and retained PVC/PV names for the implementation handover. Do not remove protection annotations or delete these volumes as part of this upgrade. + +## Completion Checklist + +Record the result of every item in the change record: + +- `PlatformLogForward/platform-default` is `Ready` and reports `LegacyESUpgradeCompleted`. +- New log, event, and audit records written after cutover can be queried from the new target. +- `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` stay absent for 60 seconds after the controlled uninstall. +- If history migration was required, the same `LegacyESMigration` reaches `Succeeded` and the migrated history passes target queries. +- The protected legacy ES PVCs and PVs are still present and bound. +- No legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workload was manually stopped, scaled, or deleted. +- If any gate failed, the change is stopped and escalated to Alauda support with the command output and resource names. + ## If a Step Is Blocked - `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. From c8243771b8568f9eb604c24dc1e2078c24e28c2e Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 14:24:54 +0800 Subject: [PATCH 05/38] docs: merge ES upgrade guide into upgrade index --- docs/en/upgrade/elasticsearch-upgrade.mdx | 502 ---------------------- docs/en/upgrade/index.mdx | 501 ++++++++++++++++++++- 2 files changed, 495 insertions(+), 508 deletions(-) delete mode 100644 docs/en/upgrade/elasticsearch-upgrade.mdx diff --git a/docs/en/upgrade/elasticsearch-upgrade.mdx b/docs/en/upgrade/elasticsearch-upgrade.mdx deleted file mode 100644 index 65d2679..0000000 --- a/docs/en/upgrade/elasticsearch-upgrade.mdx +++ /dev/null @@ -1,502 +0,0 @@ ---- -weight: 10 ---- - -# Elasticsearch Upgrade Guide - -## Introduction - -This guide upgrades a cluster that stores logs with **Alauda Container Platform Log Storage for Elasticsearch** to ACP 4.4, where new log data is written to ClickHouse or to an OpenSearch 3.7.0 cluster. - -Elasticsearch and its volumes stay in place while the platform creates the new path alongside the legacy one, so log collection is not interrupted: - -1. The platform creates the new log receiving and storage path and starts writing new log, event, and audit data to it. -2. The legacy Elasticsearch cluster and the old pipeline stay running while queued data is consumed. -3. The PlatformLogForward reports `LegacyESUpgradeCompleted` after cutover and drain complete. -4. If you need the historical data, create the migration resource and wait for `PrecaptureReady` before uninstalling the legacy plugin. -5. Confirm that the upgrade program has protected the legacy Elasticsearch volumes, then uninstall the plugin. -6. Observe the same migration resource until it succeeds, and keep the protected source volumes until validation is complete. - -:::warning -Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs and PVs before this guide tells you to. Doing it earlier can make the historical data unavailable, or prevent the source metadata snapshot from being captured. -::: - -## Scenarios - -Use this guide when the cluster matches the source and target below. - -| Item | Value | -| --- | --- | -| Source | A workload cluster with **Alauda Container Platform Log Storage for Elasticsearch** installed and running | -| Target platform | ACP 4.4 | -| Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | -| Supported source ACP versions | 4.1.x, 4.2.x, 4.3.x | - -Do not use this guide for a fresh installation, or when the cluster already stores logs in ClickHouse or OpenSearch. See [Installation](../install_log.mdx) instead. - -## Who Runs This Procedure - -This is an implementation runbook for Alauda implementation engineers or platform administrators. It is not an end-user self-service procedure. - -- Use a platform account that can access both the global management cluster and the target workload cluster. -- Use a separate terminal or kubectl context for each cluster where possible. Each command block in this guide states which cluster it runs on. -- Before each block, confirm the current context with `kubectl config current-context`. -- In a managed environment, confirm the change window and escalation path with Alauda support before you start. - -## Prerequisites - -Before you start, ensure that: - -1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. -2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. -3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - - An OpenSearch 3.7.0 cluster, or a ClickHouse cluster, sized for both the new traffic and the data that you migrate. - - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - - The target storage and Kafka connection details prepared according to their product documentation. -4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. -5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. -6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. - - -:::warning -During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Do not delete PVCs, PVs, or protection finalizers. Do not delete or recreate a `LegacyESMigration` resource. These actions can make historical data unavailable or invalidate the migration boundary. -::: - -## Upgrade Flow at a Glance - -Use this order. Each gate must pass before the next step starts. - -| Step | Where | Action | Gate to continue | -| --- | --- | --- | --- | -| 1 | Workload cluster | Prepare the target storage, Kafka, and connection Secrets | The target storage and topics are reachable | -| 2 | Workload cluster | Create `PlatformLogForward` | `Phase=Ready` and `LegacyESUpgradeCompleted` | -| 3 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | -| 4 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | -| 5 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | -| 6 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | - -## Upgrade Procedure - -### Step 1: Prepare the target storage, Kafka, and connection Secrets - -**Run on the workload cluster.** - -Create two Secrets in `cpaas-system` on the workload cluster: one for the target storage, and one for the new Kafka service. Do not overwrite or reuse the Secrets of the legacy Elasticsearch and Kafka. - -#### Target OpenSearch - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoints: "https://:9200" # OpenSearch addresses, separated by commas if there are several; put the highly available coordinator or load balancer address first - username: "" # Can be omitted when the target allows anonymous access - password: "" # Can be omitted when the target allows anonymous access - tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the worker can verify the target before writing - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- ---- -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - bootstrap: "" # Kafka addresses in host:port form, separated by commas - kafkaClusterName: "" # Kafka broker resource name; must match the actual name - username: "" # Kafka user name - password: "" # At least 32 characters on Alauda OS nodes or other FIPS-enabled hosts - sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 - topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC - topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC - topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC - tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -`endpoints` accepts several addresses. The write side uses all of them in turn, while the query side uses only the first, so put the highly available coordinator or load balancer address first. - -#### Target ClickHouse - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoint: "https://:8443" # ClickHouse address, including protocol and port - cluster: "" # ClickHouse cluster name - database: "observability" # Target database name, defaults to observability - username: "" # ClickHouse user name - password: "" # ClickHouse password - tls.ca: |- # Required when the target uses HTTPS with a private CA - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -The new Kafka service uses the same `platform-default-mq-conn` as in the OpenSearch section. - -When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` in the target storage connection Secret: the migration tool refuses to start and requires a verifiable `tls.ca` instead. - -### Step 2: Create the new data path - -**Run on the workload cluster.** - -Create one `PlatformLogForward` for your target. It must use `installMode: Fresh` and the `log.alauda.io/legacy-es-upgrade: "true"` annotation. Do not use `installMode: Adopt`. - -#### Target OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # Fixed cluster singleton name, do not change - annotations: - log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow -spec: - installMode: Fresh # Always Fresh, do not change it to Adopt - externalStorage: - type: opensearch # Target storage type - secretRef: - name: platform-default-os-conn # Target storage connection Secret created in Step 1 - namespace: cpaas-system - externalMessageQueue: - type: kafka # Message queue type, currently only kafka - secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret - namespace: cpaas-system -``` - -#### Target ClickHouse - -The `output.type` field is required for this target. - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # Fixed cluster singleton name, do not change - annotations: - log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow -spec: - installMode: Fresh # Always Fresh, do not change it to Adopt - output: - type: clickhouse # Required when the target is ClickHouse - externalStorage: - type: clickhouse # Target storage type - shards: 1 # Actual shard count of the target ClickHouse - replicas: 1 # Actual replica count of the target ClickHouse - secretRef: - name: platform-default-ch-conn # Target storage connection Secret created in Step 1 - namespace: cpaas-system - externalMessageQueue: - type: kafka # Message queue type, currently only kafka - secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret - namespace: cpaas-system -``` - -`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. - -`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. - -Save the YAML as `platform-log-forward.yaml` and apply it: - -```bash -kubectl apply -f platform-log-forward.yaml -``` - -The new data path is not available immediately. The platform first creates it, then switches the log entry point, and finally waits for the data that queued up in the old cluster to be consumed; the time this takes depends on the backlog. Watch the status until it finishes, and press `Ctrl+C` to stop: - -```bash -kubectl get platformlogforward platform-default -w -``` - -The `Phase` column changes from `Provisioning` to `Ready`, and the `Ready` column becomes `True` at the same time. Continue only after you see `Ready`. - -To follow the progress or troubleshoot, read the status conditions: - -```bash -kubectl get platformlogforward platform-default \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' -``` - -Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeCompleted`, the log entry point has switched to the new data path and the data queued in the old cluster has been consumed, so this step is complete. If the reason is `Blocked`, the `message` explains why. - -Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. - -If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform consumes the queued messages through the old path and records `LegacyKafkaDrained` when the old consumer lag reaches zero. `LegacyESUpgradeCompleted` is the gate for this procedure. - -### Step 3: Prepare the historical data migration (optional) - -**Run on the workload cluster.** - -Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. - -If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. This lets the platform select the final precapture and resolve all source PVCs while the legacy ES StatefulSet still exists. Creating the migration only after the uninstall requires explicit source PVC references and cannot use the final precapture. - -#### Target OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # Stable migration name, used by the commands below - namespace: cpaas-system -spec: - image: # Required, approved worker image with registry and tag or digest - source: - indexScope: # Required, selects the historical indices to migrate - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - target: - type: opensearch # Must match the PlatformLogForward target - secretRef: - name: platform-default-os-conn # The same connection Secret as PlatformLogForward - namespace: cpaas-system - options: - batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 - syncIntervalSeconds: 5 # Minimum wait between batches, in seconds - maxConcurrentJobs: 1 # Concurrent jobs, up to 2 for OpenSearch -``` - -#### Target ClickHouse - -Change `target.type` to `clickhouse`, point the Secret at `platform-default-ch-conn`, and keep `maxConcurrentJobs` at `1`. - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # Stable migration name, used by the commands below - namespace: cpaas-system -spec: - image: # Required, approved worker image with registry and tag or digest - source: - indexScope: # Required, selects the historical indices to migrate - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - target: - type: clickhouse # Must match the PlatformLogForward target - secretRef: - name: platform-default-ch-conn # The same connection Secret as PlatformLogForward - namespace: cpaas-system - options: - batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 - syncIntervalSeconds: 5 # Minimum wait between batches, in seconds - maxConcurrentJobs: 1 # Concurrent jobs, 1 for ClickHouse only -``` - -The standard `indexScope` categories are: - -| `indexScope` value | Data | Full index name example | -| --- | --- | --- | -| `log-workload-*` | Application and container logs | `log-workload-20260825` | -| `log-platform-*` | Platform component logs | `log-platform-20260825` | -| `log-system-*` | System logs | `log-system-20260825` | -| `log-kubernetes-*` | Kubernetes logs | `log-kubernetes-20260825` | -| `event-*` | Kubernetes events | `event-20260825` | -| `audit-*` | Audit logs | `audit-20260825` | - -If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections are immutable after creation. - -In this recommended flow, do not set `source.pvcRefs`. While the legacy StatefulSet still exists, the platform discovers every source PVC and records the result in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. Creating the migration after the uninstall requires explicit source PVC references and cannot use the final precapture. - -Apply the resource and wait for `PrecaptureReady`: - -```bash -kubectl -n cpaas-system apply -f legacy-es-migration.yaml - -# Watch the phase; press Ctrl+C to stop -kubectl -n cpaas-system get legacyesmigration platform-es-history -w - -# Show the conditions if the phase does not advance -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' -``` - -The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. - -Do not delete the migration resource or its source volumes. - -### Step 4: Uninstall the legacy Elasticsearch storage plugin - -:::warning -Run this step only when all of these conditions are true: - -- `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. -- If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. -- The upgrade program has confirmed that the legacy Elasticsearch PVC and PV protection is in place. If the PV reclaim policy is not `Retain`, or if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, stop and contact support. Do not add or modify this protection manually. -- If the source uses legacy Kafka, the platform has recorded that the old consumer lag is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. -::: - -:::warning -The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` and its `ModuleConfig` before deleting the target `ModuleInfo`. Clearing the field is required because the existing logagent still records a cross-cluster dependency on `logcenter`; it bypasses that dependency check so the target `ModuleInfo` can be removed. Do not modify or delete logagent or its `crossClusterDependency`. The temporary window affects logcenter cluster-label discovery, so run this in the approved change window. -::: - -If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected with `moduleinfo is depended by ...`, or the platform team asks you to run the controlled procedure, use the sequence below. The controlled sequence clears the discovery fields and the per-cluster install record in the order required for this upgrade. - -The critical part is a short sequence. The platform may restore the `labelCluster` field after it is cleared; that is not the durable problem. The real risk is a surviving `ClusterPluginInstance/logcenter`, which the platform can use to recreate `ModuleInfo` later. The `ModuleInfo` delete and `ClusterPluginInstance/logcenter` delete must therefore run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. - -**On the global cluster: start the critical sequence** - -Set `CLUSTER` to the workload cluster name as registered in the global cluster, then run these commands: - -```bash -set -euo pipefail - -CLUSTER= - -# 1. Resolve the target ModuleInfo. Stop if the result is ambiguous. -MODULE_INFOS="$(kubectl get moduleplugin logcenter \ - -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" -MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" -if [ "$MODULE_COUNT" -gt 1 ]; then - echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 - exit 1 -fi - -if [ "$MODULE_COUNT" -eq 1 ]; then - MODULE_INFO="$MODULE_INFOS" - MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" - MODULE_CONFIG="logcenter-${MODULE_VERSION}" - - # 2. Record the current management objects before changing them. - kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml - kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml - kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml - - # 3. Clear the discovery flags. This is also what bypasses the logagent dependency check. - kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' - kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' - - # 4. Return as soon as the delete request is accepted. Do not wait here. - kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false -else - echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." -fi -``` - -**On the workload cluster: finish the critical sequence** - -Switch the current kubectl context to the workload cluster immediately. Do not run any wait or verification first. Run: - -```bash -# 5. Delete the per-cluster install record before it can recreate ModuleInfo. -kubectl delete clusterplugininstance logcenter --ignore-not-found -``` - -**Wait for the old data path to be removed** - -Switch back to the global cluster and wait for `ModuleInfo` to disappear: - -```bash -CLUSTER= -MODULE_INFOS="$(kubectl get moduleplugin logcenter \ - -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" -MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" - -if [ -n "$MODULE_INFO" ]; then - kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m -else - echo "No ModuleInfo remains for cluster $CLUSTER." -fi -``` - -Switch to the workload cluster and wait for the legacy `AppRelease` to be removed: - -```bash -kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m -``` - -After these steps, wait 60 seconds and verify that the resources stayed gone. Each command must return no resource. - -**On the global cluster** - -```bash -kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found -``` - -**On the workload cluster** - -```bash -kubectl get clusterplugininstance logcenter --ignore-not-found -kubectl -n cpaas-system get apprelease logcenter --ignore-not-found -``` - -If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; do not manually restore them, and verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically. Do not run the workload-cluster block for that failed attempt; fix the cause and run the global sequence again. - -After the stability check passes, produce a new log, event, and audit record and confirm that it is queryable from the new target. Confirm that the platform's Razor callback and log query paths still work. Do not continue to migration verification if the new data path is unhealthy. - -### Step 5: Complete and verify the historical migration (optional) - -**Run on the workload cluster.** - -If you created `LegacyESMigration`, continue observing the same resource. Do not delete or recreate it. The phase should move from `PrecaptureReady` to `Validating`, `Running` (or `Retrying`), and finally `Succeeded`. - -```bash -kubectl -n cpaas-system get legacyesmigration platform-es-history -w - -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' -``` - -**Verification:** - -- `status.phase` is `Succeeded`. -- Every entry in `status.tasks` succeeded. -- Sample queries against the new target return the expected history for each `indexScope` category. -- The protected source PVCs and PVs are still present and bound. - -If the phase is `Blocked` or `Failed`, do not delete the migration resource, Jobs, or source volumes; read the conditions and task status, then contact support. - -If no migration was created, record the explicit decision that historical data is not required, and keep the protected source volumes until that decision is approved. - -### Step 6: Retained volumes - -The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until migration and target validation are complete. Do not remove protection annotations, delete PVCs or PVs, or remove finalizers. To release the volumes after validation, contact Alauda support or follow the separate approved cleanup procedure. - -Only the ES PVCs and PVs are retained by this upgrade. Legacy Kafka and ZooKeeper volumes are not part of the retention scope. - -Export or record the final PLF conditions, migration status, target query results, and retained PVC/PV names for the implementation handover. Do not remove protection annotations or delete these volumes as part of this upgrade. - -## Completion Checklist - -Record the result of every item in the change record: - -- `PlatformLogForward/platform-default` is `Ready` and reports `LegacyESUpgradeCompleted`. -- New log, event, and audit records written after cutover can be queried from the new target. -- `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` stay absent for 60 seconds after the controlled uninstall. -- If history migration was required, the same `LegacyESMigration` reaches `Succeeded` and the migrated history passes target queries. -- The protected legacy ES PVCs and PVs are still present and bound. -- No legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workload was manually stopped, scaled, or deleted. -- If any gate failed, the change is stopped and escalated to Alauda support with the command output and resource names. - -## If a Step Is Blocked - -- `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. -- Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. -- `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. -- `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. -- New log queries fail after uninstall: stop the migration and contact support. Do not delete the retained volumes. diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 23f8469..0d3c609 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -6,14 +6,503 @@ weight: 15 This section explains how to upgrade the Logging components of an existing ACP deployment. -Find your current deployment in the table below, then follow the matching guide. +:::info +If the cluster already stores logs in ClickHouse or OpenSearch, no upgrade guide is needed. Follow [Installation](../install_log.mdx) to install or update the plugins. +::: + +## Introduction + +This guide upgrades a cluster that stores logs with **Alauda Container Platform Log Storage for Elasticsearch** to ACP 4.4, where new log data is written to ClickHouse or to an OpenSearch 3.7.0 cluster. + +Elasticsearch and its volumes stay in place while the platform creates the new path alongside the legacy one, so log collection is not interrupted: + +1. The platform creates the new log receiving and storage path and starts writing new log, event, and audit data to it. +2. The legacy Elasticsearch cluster and the old pipeline stay running while queued data is consumed. +3. The PlatformLogForward reports `LegacyESUpgradeCompleted` after cutover and drain complete. +4. If you need the historical data, create the migration resource and wait for `PrecaptureReady` before uninstalling the legacy plugin. +5. Confirm that the upgrade program has protected the legacy Elasticsearch volumes, then uninstall the plugin. +6. Observe the same migration resource until it succeeds, and keep the protected source volumes until validation is complete. + +:::warning +Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs and PVs before this guide tells you to. Doing it earlier can make the historical data unavailable, or prevent the source metadata snapshot from being captured. +::: + +## Scenarios + +Use this guide when the cluster matches the source and target below. + +| Item | Value | +| --- | --- | +| Source | A workload cluster with **Alauda Container Platform Log Storage for Elasticsearch** installed and running | +| Target platform | ACP 4.4 | +| Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | +| Supported source Logging plugin versions | 4.2.x, 4.3.x | + +Do not use this guide for a fresh installation, or when the cluster already stores logs in ClickHouse or OpenSearch. See [Installation](../install_log.mdx) instead. + +## Who Runs This Procedure + +This is an implementation runbook for Alauda implementation engineers or platform administrators. It is not an end-user self-service procedure. + +- Use a platform account that can access both the global management cluster and the target workload cluster. +- Use a separate terminal or kubectl context for each cluster where possible. Each command block in this guide states which cluster it runs on. +- Before each block, confirm the current context with `kubectl config current-context`. +- In a managed environment, confirm the change window and escalation path with Alauda support before you start. + +## Prerequisites + +Before you start, ensure that: + +1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. +2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. +3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: + - An OpenSearch 3.7.0 cluster, or a ClickHouse cluster, sized for both the new traffic and the data that you migrate. + - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. + - The target storage and Kafka connection details prepared according to their product documentation. +4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. +5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. +6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. + + +:::warning +During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Do not delete PVCs, PVs, or protection finalizers. Do not delete or recreate a `LegacyESMigration` resource. These actions can make historical data unavailable or invalidate the migration boundary. +::: + +## Upgrade Flow at a Glance + +Use this order. Each gate must pass before the next step starts. + +| Step | Where | Action | Gate to continue | +| --- | --- | --- | --- | +| 1 | Workload cluster | Prepare the target storage, Kafka, and connection Secrets | The target storage and topics are reachable | +| 2 | Workload cluster | Create `PlatformLogForward` | `Phase=Ready` and `LegacyESUpgradeCompleted` | +| 3 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | +| 4 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | +| 5 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | +| 6 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | + +## Upgrade Procedure + +### Step 1: Prepare the target storage, Kafka, and connection Secrets + +**Run on the workload cluster.** + +Create two Secrets in `cpaas-system` on the workload cluster: one for the target storage, and one for the new Kafka service. Do not overwrite or reuse the Secrets of the legacy Elasticsearch and Kafka. + +#### Target OpenSearch + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoints: "https://:9200" # OpenSearch addresses, separated by commas if there are several; put the highly available coordinator or load balancer address first + username: "" # Can be omitted when the target allows anonymous access + password: "" # Can be omitted when the target allows anonymous access + tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the worker can verify the target before writing + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +--- +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + bootstrap: "" # Kafka addresses in host:port form, separated by commas + kafkaClusterName: "" # Kafka broker resource name; must match the actual name + username: "" # Kafka user name + password: "" # At least 32 characters on Alauda OS nodes or other FIPS-enabled hosts + sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 + topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC + topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC + topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC + tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +`endpoints` accepts several addresses. The write side uses all of them in turn, while the query side uses only the first, so put the highly available coordinator or load balancer address first. + +#### Target ClickHouse + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoint: "https://:8443" # ClickHouse address, including protocol and port + cluster: "" # ClickHouse cluster name + database: "observability" # Target database name, defaults to observability + username: "" # ClickHouse user name + password: "" # ClickHouse password + tls.ca: |- # Required when the target uses HTTPS with a private CA + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +The new Kafka service uses the same `platform-default-mq-conn` as in the OpenSearch section. + +When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` in the target storage connection Secret: the migration tool refuses to start and requires a verifiable `tls.ca` instead. + +### Step 2: Create the new data path + +**Run on the workload cluster.** + +Create one `PlatformLogForward` for your target. It must use `installMode: Fresh` and the `log.alauda.io/legacy-es-upgrade: "true"` annotation. Do not use `installMode: Adopt`. + +#### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change + annotations: + log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + externalStorage: + type: opensearch # Target storage type + secretRef: + name: platform-default-os-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +#### Target ClickHouse + +The `output.type` field is required for this target. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change + annotations: + log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + output: + type: clickhouse # Required when the target is ClickHouse + externalStorage: + type: clickhouse # Target storage type + shards: 1 # Actual shard count of the target ClickHouse + replicas: 1 # Actual replica count of the target ClickHouse + secretRef: + name: platform-default-ch-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. + +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. + +Save the YAML as `platform-log-forward.yaml` and apply it: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +The new data path is not available immediately. The platform first creates it, then switches the log entry point, and finally waits for the data that queued up in the old cluster to be consumed; the time this takes depends on the backlog. Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column changes from `Provisioning` to `Ready`, and the `Ready` column becomes `True` at the same time. Continue only after you see `Ready`. + +To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeCompleted`, the log entry point has switched to the new data path and the data queued in the old cluster has been consumed, so this step is complete. If the reason is `Blocked`, the `message` explains why. + +Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. + +If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform consumes the queued messages through the old path and records `LegacyKafkaDrained` when the old consumer lag reaches zero. `LegacyESUpgradeCompleted` is the gate for this procedure. + +### Step 3: Prepare the historical data migration (optional) + +**Run on the workload cluster.** + +Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. + +If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. This lets the platform select the final precapture and resolve all source PVCs while the legacy ES StatefulSet still exists. Creating the migration only after the uninstall requires explicit source PVC references and cannot use the final precapture. + +#### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required, approved worker image with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # Must match the PlatformLogForward target + secretRef: + name: platform-default-os-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 + syncIntervalSeconds: 5 # Minimum wait between batches, in seconds + maxConcurrentJobs: 1 # Concurrent jobs, up to 2 for OpenSearch +``` -| Current deployment | Upgrade to | Guide | +#### Target ClickHouse + +Change `target.type` to `clickhouse`, point the Secret at `platform-default-ch-conn`, and keep `maxConcurrentJobs` at `1`. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required, approved worker image with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # Must match the PlatformLogForward target + secretRef: + name: platform-default-ch-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 + syncIntervalSeconds: 5 # Minimum wait between batches, in seconds + maxConcurrentJobs: 1 # Concurrent jobs, 1 for ClickHouse only +``` + +The standard `indexScope` categories are: + +| `indexScope` value | Data | Full index name example | | --- | --- | --- | -| Logs are stored with **Alauda Container Platform Log Storage for Elasticsearch** | ACP 4.4 with ClickHouse or an OpenSearch 3.7.0 cluster | [Elasticsearch Upgrade Guide](./elasticsearch-upgrade.mdx) | +| `log-workload-*` | Application and container logs | `log-workload-20260825` | +| `log-platform-*` | Platform component logs | `log-platform-20260825` | +| `log-system-*` | System logs | `log-system-20260825` | +| `log-kubernetes-*` | Kubernetes logs | `log-kubernetes-20260825` | +| `event-*` | Kubernetes events | `event-20260825` | +| `audit-*` | Audit logs | `audit-20260825` | - +If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections are immutable after creation. -:::info -If the cluster already stores logs in ClickHouse or OpenSearch, no upgrade guide is needed. Follow [Installation](../install_log.mdx) to install or update the plugins. +In this recommended flow, do not set `source.pvcRefs`. While the legacy StatefulSet still exists, the platform discovers every source PVC and records the result in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. Creating the migration after the uninstall requires explicit source PVC references and cannot use the final precapture. + +Apply the resource and wait for `PrecaptureReady`: + +```bash +kubectl -n cpaas-system apply -f legacy-es-migration.yaml + +# Watch the phase; press Ctrl+C to stop +kubectl -n cpaas-system get legacyesmigration platform-es-history -w + +# Show the conditions if the phase does not advance +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. + +Do not delete the migration resource or its source volumes. + +### Step 4: Uninstall the legacy Elasticsearch storage plugin + +:::warning +Run this step only when all of these conditions are true: + +- `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. +- If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. +- The upgrade program has confirmed that the legacy Elasticsearch PVC and PV protection is in place. If the PV reclaim policy is not `Retain`, or if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, stop and contact support. Do not add or modify this protection manually. +- If the source uses legacy Kafka, the platform has recorded that the old consumer lag is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. ::: + +:::warning +The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` and its `ModuleConfig` before deleting the target `ModuleInfo`. Clearing the field is required because the existing logagent still records a cross-cluster dependency on `logcenter`; it bypasses that dependency check so the target `ModuleInfo` can be removed. Do not modify or delete logagent or its `crossClusterDependency`. The temporary window affects logcenter cluster-label discovery, so run this in the approved change window. +::: + +If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected with `moduleinfo is depended by ...`, or the platform team asks you to run the controlled procedure, use the sequence below. The controlled sequence clears the discovery fields and the per-cluster install record in the order required for this upgrade. + +The critical part is a short sequence. The platform may restore the `labelCluster` field after it is cleared; that is not the durable problem. The real risk is a surviving `ClusterPluginInstance/logcenter`, which the platform can use to recreate `ModuleInfo` later. The `ModuleInfo` delete and `ClusterPluginInstance/logcenter` delete must therefore run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. + +**On the global cluster: start the critical sequence** + +Set `CLUSTER` to the workload cluster name as registered in the global cluster, then run these commands: + +```bash +set -euo pipefail + +CLUSTER= + +# 1. Resolve the target ModuleInfo. Stop if the result is ambiguous. +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" +if [ "$MODULE_COUNT" -gt 1 ]; then + echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 + exit 1 +fi + +if [ "$MODULE_COUNT" -eq 1 ]; then + MODULE_INFO="$MODULE_INFOS" + MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" + MODULE_CONFIG="logcenter-${MODULE_VERSION}" + + # 2. Record the current management objects before changing them. + kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml + kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml + kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml + + # 3. Clear the discovery flags. This is also what bypasses the logagent dependency check. + kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + + # 4. Return as soon as the delete request is accepted. Do not wait here. + kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false +else + echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." +fi +``` + +**On the workload cluster: finish the critical sequence** + +Switch the current kubectl context to the workload cluster immediately. Do not run any wait or verification first. Run: + +```bash +# 5. Delete the per-cluster install record before it can recreate ModuleInfo. +kubectl delete clusterplugininstance logcenter --ignore-not-found +``` + +**Wait for the old data path to be removed** + +Switch back to the global cluster and wait for `ModuleInfo` to disappear: + +```bash +CLUSTER= +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" + +if [ -n "$MODULE_INFO" ]; then + kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No ModuleInfo remains for cluster $CLUSTER." +fi +``` + +Switch to the workload cluster and wait for the legacy `AppRelease` to be removed: + +```bash +kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +After these steps, wait 60 seconds and verify that the resources stayed gone. Each command must return no resource. + +**On the global cluster** + +```bash +kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found +``` + +**On the workload cluster** + +```bash +kubectl get clusterplugininstance logcenter --ignore-not-found +kubectl -n cpaas-system get apprelease logcenter --ignore-not-found +``` + +If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; do not manually restore them, and verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically. Do not run the workload-cluster block for that failed attempt; fix the cause and run the global sequence again. + +After the stability check passes, produce a new log, event, and audit record and confirm that it is queryable from the new target. Confirm that the platform's Razor callback and log query paths still work. Do not continue to migration verification if the new data path is unhealthy. + +### Step 5: Complete and verify the historical migration (optional) + +**Run on the workload cluster.** + +If you created `LegacyESMigration`, continue observing the same resource. Do not delete or recreate it. The phase should move from `PrecaptureReady` to `Validating`, `Running` (or `Retrying`), and finally `Succeeded`. + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history -w + +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' +``` + +**Verification:** + +- `status.phase` is `Succeeded`. +- Every entry in `status.tasks` succeeded. +- Sample queries against the new target return the expected history for each `indexScope` category. +- The protected source PVCs and PVs are still present and bound. + +If the phase is `Blocked` or `Failed`, do not delete the migration resource, Jobs, or source volumes; read the conditions and task status, then contact support. + +If no migration was created, record the explicit decision that historical data is not required, and keep the protected source volumes until that decision is approved. + +### Step 6: Retained volumes + +The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until migration and target validation are complete. Do not remove protection annotations, delete PVCs or PVs, or remove finalizers. To release the volumes after validation, contact Alauda support or follow the separate approved cleanup procedure. + +Only the ES PVCs and PVs are retained by this upgrade. Legacy Kafka and ZooKeeper volumes are not part of the retention scope. + +Export or record the final PLF conditions, migration status, target query results, and retained PVC/PV names for the implementation handover. Do not remove protection annotations or delete these volumes as part of this upgrade. + +## Completion Checklist + +Record the result of every item in the change record: + +- `PlatformLogForward/platform-default` is `Ready` and reports `LegacyESUpgradeCompleted`. +- New log, event, and audit records written after cutover can be queried from the new target. +- `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` stay absent for 60 seconds after the controlled uninstall. +- If history migration was required, the same `LegacyESMigration` reaches `Succeeded` and the migrated history passes target queries. +- The protected legacy ES PVCs and PVs are still present and bound. +- No legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workload was manually stopped, scaled, or deleted. +- If any gate failed, the change is stopped and escalated to Alauda support with the command output and resource names. + +## If a Step Is Blocked + +- `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. +- Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. +- `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. +- `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. +- New log queries fail after uninstall: stop the migration and contact support. Do not delete the retained volumes. From 13ff3a360b88e41f13f7e27ddb8ef05830e47862 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 14:33:36 +0800 Subject: [PATCH 06/38] docs: clarify OpenSearch endpoints semantics --- docs/en/upgrade/index.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 0d3c609..60efa01 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -99,7 +99,7 @@ metadata: namespace: cpaas-system type: Opaque stringData: - endpoints: "https://:9200" # OpenSearch addresses, separated by commas if there are several; put the highly available coordinator or load balancer address first + endpoints: "https://:9200" # Comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first username: "" # Can be omitted when the target allows anonymous access password: "" # Can be omitted when the target allows anonymous access tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the worker can verify the target before writing @@ -128,7 +128,7 @@ stringData: -----END CERTIFICATE----- ``` -`endpoints` accepts several addresses. The write side uses all of them in turn, while the query side uses only the first, so put the highly available coordinator or load balancer address first. +`endpoints` is a comma-separated string and accepts multiple HTTP(S) URLs. aggregate-vector and OpenSearch index initialization use the full list; the Razor query path and the historical migration worker use only the first address. Put the highly available coordinator, Service, or load balancer first, not a single data node. #### Target ClickHouse From c43497bd8ce12bcae09409e9a73e4bfabed9438b Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 14:40:39 +0800 Subject: [PATCH 07/38] docs: add ES target prerequisites and post-upgrade checks --- docs/en/upgrade/index.mdx | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 60efa01..c44c2ca 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -56,7 +56,8 @@ Before you start, ensure that: 1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. 2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - - An OpenSearch 3.7.0 cluster, or a ClickHouse cluster, sized for both the new traffic and the data that you migrate. + - For OpenSearch 3.7.0, every OpenSearch node must have the `analysis-ik` plugin in the matching 3.7.0 version. Without it, log queries fail even when writes succeed. Confirm it through the OpenSearch plugin list and an `ik_smart` analyze check before cutover. The target connection user must be able to manage index templates and ISM policies and read/write the log data. + - For ClickHouse, use a supported cluster with `ReplicatedMergeTree` and Keeper/ZooKeeper. The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target user must be able to create/update the target database schema, read/write data, and run `SYSTEM DROP DNS CACHE` for the Razor cleaner. - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - The target storage and Kafka connection details prepared according to their product documentation. 4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. @@ -99,7 +100,7 @@ metadata: namespace: cpaas-system type: Opaque stringData: - endpoints: "https://:9200" # Comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first + endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first username: "" # Can be omitted when the target allows anonymous access password: "" # Can be omitted when the target allows anonymous access tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the worker can verify the target before writing @@ -114,10 +115,10 @@ metadata: namespace: cpaas-system type: Opaque stringData: - bootstrap: "" # Kafka addresses in host:port form, separated by commas - kafkaClusterName: "" # Kafka broker resource name; must match the actual name - username: "" # Kafka user name - password: "" # At least 32 characters on Alauda OS nodes or other FIPS-enabled hosts + bootstrap: "" # Required; Kafka addresses in host:port form, separated by commas + kafkaClusterName: "" # Required; Kafka broker resource name; must match the actual name + username: "" # Required; Kafka user name + password: "" # Required; at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC @@ -128,7 +129,7 @@ stringData: -----END CERTIFICATE----- ``` -`endpoints` is a comma-separated string and accepts multiple HTTP(S) URLs. aggregate-vector and OpenSearch index initialization use the full list; the Razor query path and the historical migration worker use only the first address. Put the highly available coordinator, Service, or load balancer first, not a single data node. +`endpoints` is a comma-separated string and accepts multiple HTTP(S) URLs. aggregate-vector and OpenSearch index initialization use the full list; the Razor query path and the historical migration worker use only the first address. Put the highly available coordinator, Service, or load balancer first, not a single data node, and do not add leading whitespace before the first URL. #### Target ClickHouse @@ -140,8 +141,8 @@ metadata: namespace: cpaas-system type: Opaque stringData: - endpoint: "https://:8443" # ClickHouse address, including protocol and port - cluster: "" # ClickHouse cluster name + endpoint: "https://:8443" # Required; ClickHouse address, including protocol and port + cluster: "replicated" # Must match the ClickHouse cluster name; ACP baseline default is replicated database: "observability" # Target database name, defaults to observability username: "" # ClickHouse user name password: "" # ClickHouse password @@ -242,7 +243,7 @@ Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeComple Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. -If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform consumes the queued messages through the old path and records `LegacyKafkaDrained` when the old consumer lag reaches zero. `LegacyESUpgradeCompleted` is the gate for this procedure. +If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The legacy lanaya path continues consuming the queued messages and writes them to legacy Elasticsearch; the platform observes the old consumer lag and records `LegacyKafkaDrained` when it reaches zero. `LegacyESUpgradeCompleted` is the gate for this procedure. ### Step 3: Prepare the historical data migration (optional) @@ -351,12 +352,12 @@ Run this step only when all of these conditions are true: - `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. - If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. -- The upgrade program has confirmed that the legacy Elasticsearch PVC and PV protection is in place. If the PV reclaim policy is not `Retain`, or if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, stop and contact support. Do not add or modify this protection manually. +- The upgrade program has confirmed that the legacy Elasticsearch volume-protection gate is complete, including the PVC/PV protection fields and the protection webhooks. If the PV reclaim policy is not `Retain`, if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, or if the platform reports a protection gate failure, stop and contact support. Do not add or modify this protection manually. - If the source uses legacy Kafka, the platform has recorded that the old consumer lag is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. ::: :::warning -The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` and its `ModuleConfig` before deleting the target `ModuleInfo`. Clearing the field is required because the existing logagent still records a cross-cluster dependency on `logcenter`; it bypasses that dependency check so the target `ModuleInfo` can be removed. Do not modify or delete logagent or its `crossClusterDependency`. The temporary window affects logcenter cluster-label discovery, so run this in the approved change window. +The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` and its `ModuleConfig` before deleting the target `ModuleInfo`. Clearing the field is required when the existing logagent still records a cross-cluster dependency on `logcenter`; it bypasses that dependency check so the target `ModuleInfo` can be removed. Do not modify or delete logagent or its `crossClusterDependency`. The temporary window affects logcenter cluster-label discovery, so run this in the approved change window. ::: If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected with `moduleinfo is depended by ...`, or the platform team asks you to run the controlled procedure, use the sequence below. The controlled sequence clears the discovery fields and the per-cluster install record in the order required for this upgrade. @@ -447,6 +448,7 @@ kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name= ```bash kubectl get clusterplugininstance logcenter --ignore-not-found kubectl -n cpaas-system get apprelease logcenter --ignore-not-found +kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found ``` If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; do not manually restore them, and verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically. Do not run the workload-cluster block for that failed attempt; fix the cause and run the global sequence again. @@ -495,8 +497,8 @@ Record the result of every item in the change record: - New log, event, and audit records written after cutover can be queried from the new target. - `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` stay absent for 60 seconds after the controlled uninstall. - If history migration was required, the same `LegacyESMigration` reaches `Succeeded` and the migrated history passes target queries. -- The protected legacy ES PVCs and PVs are still present and bound. -- No legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workload was manually stopped, scaled, or deleted. +- The legacy ES StatefulSet and old logcenter workloads are gone, while the protected legacy ES PVCs and PVs are still present and bound. +- No legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workload was manually stopped, scaled, or deleted outside the controlled uninstall. - If any gate failed, the change is stopped and escalated to Alauda support with the command output and resource names. ## If a Step Is Blocked From 575e2de8880c1f66f605d5edb04e20911a0bf764 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 16:04:12 +0800 Subject: [PATCH 08/38] docs: record ES upgrade validation findings --- docs/en/upgrade/index.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index c44c2ca..20f52d8 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -57,7 +57,7 @@ Before you start, ensure that: 2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - For OpenSearch 3.7.0, every OpenSearch node must have the `analysis-ik` plugin in the matching 3.7.0 version. Without it, log queries fail even when writes succeed. Confirm it through the OpenSearch plugin list and an `ik_smart` analyze check before cutover. The target connection user must be able to manage index templates and ISM policies and read/write the log data. - - For ClickHouse, use a supported cluster with `ReplicatedMergeTree` and Keeper/ZooKeeper. The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target user must be able to create/update the target database schema, read/write data, and run `SYSTEM DROP DNS CACHE` for the Razor cleaner. + - For ClickHouse, use a supported cluster with `ReplicatedMergeTree` and Keeper/ZooKeeper. The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target user must be able to create/update the target database schema, read/write data, and run `SYSTEM DROP DNS CACHE` for the Razor cleaner. Ensure the target database exists before you apply the PlatformLogForward; if the new Razor remains unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - The target storage and Kafka connection details prepared according to their product documentation. 4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. @@ -343,6 +343,8 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. +Use the migration worker image supplied with the same ACP 4.4 Logging package. Do not reuse an older migration worker tag: an older worker can finish writing and verifying the target data but fail to terminate, leaving the migration resource in `Running`. If the target data is complete but the resource stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. + Do not delete the migration resource or its source volumes. ### Step 4: Uninstall the legacy Elasticsearch storage plugin @@ -507,4 +509,6 @@ Record the result of every item in the change record: - Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. - `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. - `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. +- New Razor remains unready with `UNKNOWN_DATABASE`: create the target ClickHouse database and wait for reconcile. Do not delete the PLF or the target Secrets. +- Migration data is verified but the resource remains `Running`: do not delete the migration resource or source volumes. Use the approved worker image for the current Logging package and contact support if the resource still does not complete. - New log queries fail after uninstall: stop the migration and contact support. Do not delete the retained volumes. From 7912800e4e64a8ee9b5e7a71c8a87a8464e2dd57 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 18:36:56 +0800 Subject: [PATCH 09/38] docs: clarify target writes during ES migration --- docs/en/upgrade/index.mdx | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 20f52d8..6aee7bc 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -343,6 +343,10 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. +:::warning +New log, event, and audit data continues to be written to the target after cutover. This is expected. Do not delete or recreate target indices, and do not stop the new data path just to make source and target counts equal. The migration compares against the captured source baseline and requires the target to contain at least the migrated history plus any new data. +::: + Use the migration worker image supplied with the same ACP 4.4 Logging package. Do not reuse an older migration worker tag: an older worker can finish writing and verifying the target data but fail to terminate, leaving the migration resource in `Running`. If the target data is complete but the resource stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. Do not delete the migration resource or its source volumes. @@ -507,6 +511,7 @@ Record the result of every item in the change record: - `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. - Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. +- Target document count mismatch: keep the new data path running and do not delete target indices. The target can contain more documents than the source baseline because new data is written after cutover. If the migration still fails, contact support with the source baseline and target count evidence. - `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. - `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. - New Razor remains unready with `UNKNOWN_DATABASE`: create the target ClickHouse database and wait for reconcile. Do not delete the PLF or the target Secrets. From 82052a1340726632ff17e706623e40d7dc2340b2 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 18:49:59 +0800 Subject: [PATCH 10/38] Revert "docs: clarify target writes during ES migration" This reverts commit 7912800e4e64a8ee9b5e7a71c8a87a8464e2dd57. --- docs/en/upgrade/index.mdx | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 6aee7bc..20f52d8 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -343,10 +343,6 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. -:::warning -New log, event, and audit data continues to be written to the target after cutover. This is expected. Do not delete or recreate target indices, and do not stop the new data path just to make source and target counts equal. The migration compares against the captured source baseline and requires the target to contain at least the migrated history plus any new data. -::: - Use the migration worker image supplied with the same ACP 4.4 Logging package. Do not reuse an older migration worker tag: an older worker can finish writing and verifying the target data but fail to terminate, leaving the migration resource in `Running`. If the target data is complete but the resource stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. Do not delete the migration resource or its source volumes. @@ -511,7 +507,6 @@ Record the result of every item in the change record: - `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. - Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. -- Target document count mismatch: keep the new data path running and do not delete target indices. The target can contain more documents than the source baseline because new data is written after cutover. If the migration still fails, contact support with the source baseline and target count evidence. - `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. - `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. - New Razor remains unready with `UNKNOWN_DATABASE`: create the target ClickHouse database and wait for reconcile. Do not delete the PLF or the target Secrets. From 9544ae22d74501ba57163da5c16230852f513de7 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 14 Sep 2026 19:18:52 +0800 Subject: [PATCH 11/38] docs: refine ES upgrade preparation, alignment, and acceptance - add target storage preparation with capacity-planning reference and preflight checklist - align target settings with the deployed operator CRs (ClickHouseInstallation, Kafka/KafkaNodePool/KafkaTopic, OpenSearch templates) - treat analysis-ik as optional and base acceptance on resource status - keep product-exposed options, index examples, and migration status details --- docs/en/upgrade/index.mdx | 115 +++++++++++++++++++++++++------------- 1 file changed, 77 insertions(+), 38 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 20f52d8..69a22ec 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -56,11 +56,11 @@ Before you start, ensure that: 1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. 2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - - For OpenSearch 3.7.0, every OpenSearch node must have the `analysis-ik` plugin in the matching 3.7.0 version. Without it, log queries fail even when writes succeed. Confirm it through the OpenSearch plugin list and an `ik_smart` analyze check before cutover. The target connection user must be able to manage index templates and ISM policies and read/write the log data. - - For ClickHouse, use a supported cluster with `ReplicatedMergeTree` and Keeper/ZooKeeper. The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target user must be able to create/update the target database schema, read/write data, and run `SYSTEM DROP DNS CACHE` for the Razor cleaner. Ensure the target database exists before you apply the PlatformLogForward; if the new Razor remains unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. + - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches use the OpenSearch standard analyzer: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. + - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - The target storage and Kafka connection details prepared according to their product documentation. -4. If you migrate historical data, you have the approved migration worker image. The image is not part of the Operator package; specify the complete version, tag, or digest. +4. If you migrate historical data, you have the migration image provided for this release, with the complete registry, tag, or digest. Do not reuse an image from an earlier version. 5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. 6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. @@ -69,6 +69,45 @@ Before you start, ensure that: During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Do not delete PVCs, PVs, or protection finalizers. Do not delete or recreate a `LegacyESMigration` resource. These actions can make historical data unavailable or invalidate the migration boundary. ::: +## Target Storage Preparation + +The operator does not create the external OpenSearch/ClickHouse cluster or the new Kafka service. Create them separately, then provide the connection details through the Secrets in Step 1. + +Plan the size with [Log Component Capacity Planning](../architecture/capacity_planning.mdx), and do not plan below the current logcenter deployment. The target must hold the migrated history plus new traffic. The reference disk configuration in that guide is `6000 IOPS` and `250 MB/s` read/write on dedicated SSD mounts; if the actual storage is weaker, move to a larger profile. + +The legacy logcenter values below are the current deployment baseline, not target settings. They are `chart-alauda-log-center` defaults and can differ per site; check the deployed values before you plan: + +| Legacy logcenter chart setting | Typical baseline | What to prepare for the target | +| --- | --- | --- | +| `elasticsearch.storage.node_size`, `node_replicas` | 200 Gi per data node; `node_replicas` sets the data-node count (1 by default; the small-scale profile uses 3) | Size the target storage for the historical data plus new traffic and the target HA policy | +| `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | Use independent storage for the new target; the legacy local path is not reused | +| `logging.esReplicas`, `logging.shards` | Index-level settings: 1 replica, per-type shard counts | OpenSearch only: these are the baseline for the index templates; override the template if the target needs different values. They do not map to ClickHouse — ClickHouse uses `externalStorage.shards` and `replicas` for the cluster topology (see the alignment table below) | +| `kafka.retention_hours` | 48 hours | Set the same retention in the target `KafkaTopic` CRs; size the Kafka brokers through your Kafka deployment, not from this legacy chart value | +| `logging.ttl` | Logs 7 days; events/audits 180 days; metering 540 days | `PlatformLogForward` applies the same TTLs to the target; plan capacity with the retention you configure there | + +The operator applies the TTL and, for ClickHouse, the shard and replica values from the `PlatformLogForward` spec, so the prepared cluster must be able to satisfy them. The chart does not cover the target node size: size ClickHouse with the capacity-planning profiles, and size OpenSearch with your OpenSearch deployment sizing using the same data volume and throughput inputs. + +### Aligning the target with the PlatformLogForward + +Read the target settings from the CRs that are actually deployed in the environment, not from this guide: + +| Target | Where to read the deployed settings | What must line up | +| --- | --- | --- | +| ClickHouse | `ClickHouseInstallation` CR: `spec.configuration.clusters[].layout.shardsCount` and `replicasCount` | Set `spec.externalStorage.shards` and `replicas` in the `PlatformLogForward` to the values declared in the CHI CR, and set the Secret `cluster` value to the cluster name used by `ON CLUSTER`. The data path expands by these two values, so a mismatch writes to the wrong replica set. | +| Kafka | `Kafka` CR (`spec.kafka.config`), `KafkaNodePool` (or `Kafka.spec.kafka` in older layouts) for broker count and storage, `KafkaTopic` CRs, and `KafkaUser` CR | Keep the Secret topic names (`topics.log`, `topics.event`, `topics.audit`) equal to `KafkaTopic.spec.topicName` and to the topic names in the `KafkaUser` ACLs, and keep `kafkaClusterName` equal to the Kafka CR name. Set partitions, replication factor, and retention in the `KafkaTopic` CRs (or the equivalent topic configuration on your Kafka deployment); the replication factor cannot exceed the number of brokers. The broker and topic maximum message size must accept the audit batches — the Kafka default is 1 MiB, which is too small; the reference CRs use 10 MiB. Keep `auto.create.topics.enable` disabled so a misnamed topic is not created automatically. | +| OpenSearch | The deployed OpenSearch cluster (its operator CR or the manifests that run it) and the applied index templates (`GET /_index_template`) | Index shards and replicas are not controlled by the `PlatformLogForward`. The platform applies low-priority templates with 1 shard and 1 replica; if the production node count or HA policy needs different values, apply a higher-priority composable template before cutover and confirm the result with `GET /_index_template`. | + +## Preflight Checklist + +| Check | Expected | +| --- | --- | +| Source | Legacy ES, Kafka, ZooKeeper, lanaya, and Razor are running; `logcenter` is `Running` | +| Target storage | OpenSearch 3.7.0 or the supported ClickHouse topology is reachable; database/Keeper requirements are met | +| Target Kafka | New bootstrap is reachable; topics and Logging user access are ready | +| Connection Secrets | Required keys are present and valid; legacy Secrets are not reused | +| Access and change window | Platform administrator can reach global and workload clusters; support path and change window are confirmed | +| Historical migration | The migration image for this release is available; the target account can create and write the required schema and data | + ## Upgrade Flow at a Glance Use this order. Each gate must pass before the next step starts. @@ -129,7 +168,7 @@ stringData: -----END CERTIFICATE----- ``` -`endpoints` is a comma-separated string and accepts multiple HTTP(S) URLs. aggregate-vector and OpenSearch index initialization use the full list; the Razor query path and the historical migration worker use only the first address. Put the highly available coordinator, Service, or load balancer first, not a single data node, and do not add leading whitespace before the first URL. +`endpoints` accepts multiple comma-separated HTTP(S) URLs. Some data paths use only the first address, so put the highly available load balancer or coordinator endpoint first, not a single data node, and do not add leading whitespace before the first URL. #### Target ClickHouse @@ -154,7 +193,7 @@ stringData: The new Kafka service uses the same `platform-default-mq-conn` as in the OpenSearch section. -When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` in the target storage connection Secret: the migration tool refuses to start and requires a verifiable `tls.ca` instead. +When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` in the target storage connection Secret; provide `tls.ca` instead so the migration can verify the target. ### Step 2: Create the new data path @@ -243,7 +282,7 @@ Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeComple Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. -If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The legacy lanaya path continues consuming the queued messages and writes them to legacy Elasticsearch; the platform observes the old consumer lag and records `LegacyKafkaDrained` when it reaches zero. `LegacyESUpgradeCompleted` is the gate for this procedure. +If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform drains the queued data through the legacy path automatically and records `LegacyKafkaDrained` when the old consumer lag reaches zero; `LegacyESUpgradeCompleted` is the gate for this procedure. ### Step 3: Prepare the historical data migration (optional) @@ -251,7 +290,7 @@ If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeep Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. -If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. This lets the platform select the final precapture and resolve all source PVCs while the legacy ES StatefulSet still exists. Creating the migration only after the uninstall requires explicit source PVC references and cannot use the final precapture. +If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin, so the platform can capture the final source state and resolve the source volumes automatically. Creating the migration after the uninstall requires explicit source volume references and cannot use the final capture. #### Target OpenSearch @@ -262,7 +301,7 @@ metadata: name: platform-es-history # Stable migration name, used by the commands below namespace: cpaas-system spec: - image: # Required, approved worker image with registry and tag or digest + image: # Required; migration image provided for this release, with registry and tag or digest source: indexScope: # Required, selects the historical indices to migrate - "log-workload-*" @@ -284,7 +323,7 @@ spec: #### Target ClickHouse -Change `target.type` to `clickhouse`, point the Secret at `platform-default-ch-conn`, and keep `maxConcurrentJobs` at `1`. +Change `target.type` to `clickhouse` and point the Secret at `platform-default-ch-conn`. ```yaml apiVersion: log.alauda.io/v1alpha1 @@ -293,7 +332,7 @@ metadata: name: platform-es-history # Stable migration name, used by the commands below namespace: cpaas-system spec: - image: # Required, approved worker image with registry and tag or digest + image: # Required; migration image provided for this release, with registry and tag or digest source: indexScope: # Required, selects the historical indices to migrate - "log-workload-*" @@ -324,9 +363,9 @@ The standard `indexScope` categories are: | `event-*` | Kubernetes events | `event-20260825` | | `audit-*` | Audit logs | `audit-20260825` | -If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections are immutable after creation. +If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections cannot be changed after creation. The migration options above show the defaults; adjust them only when your migration plan requires it. -In this recommended flow, do not set `source.pvcRefs`. While the legacy StatefulSet still exists, the platform discovers every source PVC and records the result in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. Creating the migration after the uninstall requires explicit source PVC references and cannot use the final precapture. +Do not set `source.pvcRefs` in this recommended flow: the platform discovers the source volumes automatically while the legacy Elasticsearch StatefulSet still exists and records them in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. Apply the resource and wait for `PrecaptureReady`: @@ -341,11 +380,13 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` -The migration does not mount a source volume or copy data while the legacy ES StatefulSet is still running. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` when observing an existing completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. +Data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. -Use the migration worker image supplied with the same ACP 4.4 Logging package. Do not reuse an older migration worker tag: an older worker can finish writing and verifying the target data but fail to terminate, leaving the migration resource in `Running`. If the target data is complete but the resource stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. +:::info +The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. +::: -Do not delete the migration resource or its source volumes. +Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. ### Step 4: Uninstall the legacy Elasticsearch storage plugin @@ -354,17 +395,17 @@ Run this step only when all of these conditions are true: - `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. - If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. -- The upgrade program has confirmed that the legacy Elasticsearch volume-protection gate is complete, including the PVC/PV protection fields and the protection webhooks. If the PV reclaim policy is not `Retain`, if the PVC and PV lack `helm.sh/resource-policy=keep`, `skip-sync=true`, or the protection label, or if the platform reports a protection gate failure, stop and contact support. Do not add or modify this protection manually. -- If the source uses legacy Kafka, the platform has recorded that the old consumer lag is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. +- The platform reports that protection for the legacy Elasticsearch volumes (PVC/PV) is complete. Do not modify or remove this protection; if the platform reports a protection failure, stop and contact support. +- If the source uses legacy Kafka, the platform has confirmed that all queued data is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. ::: :::warning -The controlled sequence clears `spec.labelCluster` on `ModulePlugin/logcenter` and its `ModuleConfig` before deleting the target `ModuleInfo`. Clearing the field is required when the existing logagent still records a cross-cluster dependency on `logcenter`; it bypasses that dependency check so the target `ModuleInfo` can be removed. Do not modify or delete logagent or its `crossClusterDependency`. The temporary window affects logcenter cluster-label discovery, so run this in the approved change window. +This sequence temporarily clears platform discovery fields for `logcenter` so the legacy plugin can be removed. Run it only in the approved change window, and do not modify or delete logagent or its dependencies. ::: -If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected with `moduleinfo is depended by ...`, or the platform team asks you to run the controlled procedure, use the sequence below. The controlled sequence clears the discovery fields and the per-cluster install record in the order required for this upgrade. +If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected, or the platform team asks you to run the controlled procedure, use the sequence below. -The critical part is a short sequence. The platform may restore the `labelCluster` field after it is cleared; that is not the durable problem. The real risk is a surviving `ClusterPluginInstance/logcenter`, which the platform can use to recreate `ModuleInfo` later. The `ModuleInfo` delete and `ClusterPluginInstance/logcenter` delete must therefore run back-to-back. Do not wait for `ModuleInfo` to disappear, check `AppRelease`, or do anything else between them. +The two deletes must run back-to-back: do not wait for `ModuleInfo` to disappear, check `AppRelease`, or perform any other check between them. **On the global cluster: start the critical sequence** @@ -453,9 +494,9 @@ kubectl -n cpaas-system get apprelease logcenter --ignore-not-found kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found ``` -If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. The `labelCluster` fields may be rewritten to `true`; do not manually restore them, and verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, the platform normally restores `labelCluster` automatically. Do not run the workload-cluster block for that failed attempt; fix the cause and run the global sequence again. +If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. Do not restore the cleared discovery fields manually; verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, do not run the workload-cluster block for that attempt: fix the cause and run the global sequence again. -After the stability check passes, produce a new log, event, and audit record and confirm that it is queryable from the new target. Confirm that the platform's Razor callback and log query paths still work. Do not continue to migration verification if the new data path is unhealthy. +After the stability check passes, produce new log, event, and audit records and confirm that they can be queried from the new target. Do not continue to migration verification if the new data path is unhealthy. ### Step 5: Complete and verify the historical migration (optional) @@ -479,7 +520,7 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ - Sample queries against the new target return the expected history for each `indexScope` category. - The protected source PVCs and PVs are still present and bound. -If the phase is `Blocked` or `Failed`, do not delete the migration resource, Jobs, or source volumes; read the conditions and task status, then contact support. +If the phase is `Blocked` or `Failed`, do not delete the migration resource, Jobs, or source volumes; contact support. If no migration was created, record the explicit decision that historical data is not required, and keep the protected source volumes until that decision is approved. @@ -493,22 +534,20 @@ Export or record the final PLF conditions, migration status, target query result ## Completion Checklist -Record the result of every item in the change record: +| Status | Expected | +| --- | --- | +| `PlatformLogForward/platform-default` | `Phase=Ready` and `LegacyESUpgradeCompleted` | +| `LegacyESMigration/platform-es-history` (if created) | `Phase=Succeeded` | +| New logging query | A new log, event, and audit record is returned by the new query path | +| Legacy plugin removal | `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` are absent, and the old ES workload is gone | +| Source volumes | Protected legacy ES PVCs and PVs remain present | -- `PlatformLogForward/platform-default` is `Ready` and reports `LegacyESUpgradeCompleted`. -- New log, event, and audit records written after cutover can be queried from the new target. -- `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` stay absent for 60 seconds after the controlled uninstall. -- If history migration was required, the same `LegacyESMigration` reaches `Succeeded` and the migrated history passes target queries. -- The legacy ES StatefulSet and old logcenter workloads are gone, while the protected legacy ES PVCs and PVs are still present and bound. -- No legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workload was manually stopped, scaled, or deleted outside the controlled uninstall. -- If any gate failed, the change is stopped and escalated to Alauda support with the command output and resource names. +If any status is not as expected, stop and contact Alauda support. Do not delete the migration resource, source volumes, or target data to work around a failure. ## If a Step Is Blocked -- `moduleinfo is depended by ...`: stop. Do not modify or delete logagent. Confirm that both `ModulePlugin` and `ModuleConfig` had `labelCluster` cleared before the `ModuleInfo` deletion; if the error remains, contact support. -- Migration phase `Blocked`: read the condition and task status. Do not delete or recreate the migration resource. -- `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, then repeat the 60-second stability check. -- `AppRelease/logcenter` does not disappear: do not remove finalizers. Check the platform conditions and contact support. -- New Razor remains unready with `UNKNOWN_DATABASE`: create the target ClickHouse database and wait for reconcile. Do not delete the PLF or the target Secrets. -- Migration data is verified but the resource remains `Running`: do not delete the migration resource or source volumes. Use the approved worker image for the current Logging package and contact support if the resource still does not complete. -- New log queries fail after uninstall: stop the migration and contact support. Do not delete the retained volumes. +- The platform rejects the plugin removal with `moduleinfo is depended by ...`: stop, do not modify logagent, repeat the controlled sequence, and contact support if it still fails. +- Migration is `Blocked`, `Failed`, or stays `Running`: do not delete the migration resource, Jobs, source volumes, or target data; contact support. +- `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, and repeat the 60-second stability check. +- `AppRelease/logcenter` does not disappear: do not remove finalizers; contact support. +- New log queries fail after uninstall: stop and contact support. Do not delete the retained volumes. From 9f554484135182a4326ad6cb0b925d923d4f824d Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 08:51:48 +0800 Subject: [PATCH 12/38] docs: add Chinese translation for the ES upgrade guide Translate docs/en/upgrade/index.mdx to Simplified Chinese under docs/zh, with sourceSHA for the translation pipeline. Code blocks stay identical; only comments are localized. --- docs/zh/upgrade/index.mdx | 553 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 553 insertions(+) create mode 100644 docs/zh/upgrade/index.mdx diff --git a/docs/zh/upgrade/index.mdx b/docs/zh/upgrade/index.mdx new file mode 100644 index 0000000..ad4af65 --- /dev/null +++ b/docs/zh/upgrade/index.mdx @@ -0,0 +1,553 @@ +--- +weight: 15 +sourceSHA: ae9e0e3a793eebf71d8cd10c294b316313c8dc578a27a9a099cf1b94dee84ae7 +--- + +# 升级 + +本文介绍如何升级现有 ACP 部署中的日志组件。 + +:::info +如果集群已使用 ClickHouse 或 OpenSearch 存储日志，则无需执行本升级指南，请参考[安装](https://docs.alauda.cn/logging-service/4.4/install_log.html)完成插件安装或更新。 +::: + +## 简介 + +本指南用于将使用 **Alauda Container Platform Log Storage for Elasticsearch** 存储日志的集群升级到 ACP 4.4。升级后，新的日志数据将写入 ClickHouse 或 OpenSearch 3.7.0 集群。 + +升级过程中 Elasticsearch 及其存储卷保持不动，平台会在旧链路旁边创建新链路，因此日志采集不会中断： + +1. 平台创建新的日志接收与存储链路，并开始将新的日志、事件和审计数据写入该链路。 +2. 旧 Elasticsearch 集群和旧链路保持运行，直到队列中的数据消费完成。 +3. 切换和排空完成后，PlatformLogForward 会报告 `LegacyESUpgradeCompleted`。 +4. 如果需要保留历史数据，请创建迁移资源，并在卸载旧插件前等待其进入 `PrecaptureReady`。 +5. 确认升级程序已完成旧 Elasticsearch 存储卷保护，然后卸载旧插件。 +6. 持续观察同一个迁移资源直到其成功，并在校验完成前保留受保护的源端存储卷。 + +:::warning +在本指南明确要求之前，请勿卸载 Elasticsearch 存储插件、停止旧数据链路，或删除其 PVC 和 PV。提前执行这些操作可能导致历史数据不可用，或导致无法捕获源端元数据快照。 +::: + +## 适用场景 + +当集群符合下面的源端和目标端条件时，请使用本指南。 + +| 项目 | 值 | +| --- | --- | +| 源端 | 已安装并运行 **Alauda Container Platform Log Storage for Elasticsearch** 的业务集群 | +| 目标平台 | ACP 4.4 | +| 目标存储 | 单独准备的 ClickHouse 或 OpenSearch 3.7.0 集群 | +| 支持的源端日志插件版本 | 4.2.x、4.3.x | + +全新安装，或集群已使用 ClickHouse / OpenSearch 存储日志时，请勿使用本指南，请参考[安装](https://docs.alauda.cn/logging-service/4.4/install_log.html)。 + +## 适用人员 + +本文是面向 Alauda 实施工程师或平台管理员的实施手册，不是最终用户的自助操作流程。 + +- 请使用可同时访问全局管理集群和目标业务集群的平台账号。 +- 尽量为每个集群使用独立的终端或 kubectl context。本文每个命令块都说明了在哪个集群执行。 +- 执行每个命令块前，请先用 `kubectl config current-context` 确认当前 context。 +- 在托管环境中，开始操作前请与 Alauda 支持确认变更窗口和升级路径。 + +## 前置条件 + +开始前，请确保： + +1. ACP 4.4 平台升级已完成，且日志组件可以升级。请在本流程中升级日志控制组件，并保持 **Alauda Container Platform Log Storage for Elasticsearch** 处于安装状态。 +2. 已从 **Alauda Cloud** 下载 ACP 4.4 日志插件包，且该插件包已上传到集群的插件市场。 +3. 已单独准备本次升级所需的目标存储和消息队列。本次升级不会复用 Elasticsearch 插件自带的存储和 Kafka，需要准备： + - 对于 OpenSearch 3.7.0，`analysis-ik` 插件可选。未安装时，日志查询使用 OpenSearch 内置的 standard 分词器：查询仍可正常执行，但中文全文检索效果会下降。如果现场需要中文检索质量，请在每个节点安装与 3.7.0 匹配的该插件。连接账号需要具备索引模板和生命周期策略的管理权限，以及日志数据的读写权限。 + - 对于 ClickHouse，请使用支持 `ReplicatedMergeTree` 和 Keeper/ZooKeeper 的复制集群。连接 Secret 中的 `cluster` 值必须与 ClickHouse 集群名一致；目标数据库必须在创建 PlatformLogForward 之前存在，如果新链路因 `UNKNOWN_DATABASE` 一直未就绪，请先创建数据库并等待下一次调谐。连接账号需要具备建表和改表、数据读写权限，以及执行 `SYSTEM DROP DNS CACHE` 的权限。 + - 新的 Kafka 服务，并已创建 `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC` 和 `ALAUDA_AUDIT_TOPIC` 三个 topic，日志 Kafka 用户已获得这些 topic 及相关消费组的访问权限。 + - 按各自产品文档准备好的目标存储和 Kafka 连接信息。 +4. 如果需要迁移历史数据，已取得本版本提供的迁移镜像，并包含完整的 registry、tag 或 digest。请勿复用旧版本的镜像。 +5. 已获得批准的变更窗口，且平台管理员可同时访问全局管理集群和目标业务集群。在托管环境中，请与 Alauda 支持协同。 +6. 旧 Elasticsearch、Kafka、ZooKeeper、lanaya 和 Razor 工作负载仍在运行。在本指南要求之前，请勿停止、缩容或删除它们。 + +:::warning +升级期间，请勿停止、缩容或删除旧 Elasticsearch、Kafka、ZooKeeper、lanaya 或 Razor 工作负载；请勿删除 PVC、PV 或保护 finalizer；请勿删除或重建 `LegacyESMigration` 资源。这些操作可能导致历史数据不可用，或破坏迁移边界。 +::: + +## 目标存储准备 + +Operator 不会创建外部 OpenSearch/ClickHouse 集群和新的 Kafka 服务，需要单独创建，并通过步骤 1 中的 Secret 提供连接信息。 + +请参考[日志组件容量规划](https://docs.alauda.cn/logging-service/4.4/architecture/capacity_planning.html)规划容量，且不要低于当前 logcenter 部署的规格。目标存储需要能够容纳迁移的历史数据和新产生的数据。该指南中的参考磁盘配置为 `6000 IOPS`、`250 MB/s` 读写、独立 SSD 挂载；如果实际存储性能低于该配置，请选择更大规格。 + +下表中的旧 logcenter 参数是当前部署的基线，而不是目标端配置。它们来自 `chart-alauda-log-center` 的默认值，不同现场可能不同，规划前请先核对实际部署值： + +| 旧 logcenter chart 参数 | 典型基线 | 需要为目标端准备什么 | +| --- | --- | --- | +| `elasticsearch.storage.node_size`、`node_replicas` | 每个数据节点 200 Gi；`node_replicas` 表示数据节点数量（默认 1，小规模档位为 3） | 按历史数据量、新增数据量和目标端 HA 策略规划目标存储 | +| `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | 目标端使用独立存储，不复用旧的本地路径 | +| `logging.esReplicas`、`logging.shards` | 索引级配置：1 副本、按类型设置分片数 | 仅适用于 OpenSearch：作为索引模板的基线；如果目标端需要不同的值，请覆盖模板。它们不适用于 ClickHouse——ClickHouse 使用 `externalStorage.shards` 和 `replicas` 描述集群拓扑（见下方对齐表） | +| `kafka.retention_hours` | 48 小时 | 在目标 `KafkaTopic` CR 中设置相同的保留时间；Kafka broker 的容量请按你的 Kafka 部署规划，不要沿用该旧 chart 参数 | +| `logging.ttl` | 日志 7 天；事件/审计 180 天；计量 540 天 | PlatformLogForward 会把相同的 TTL 下发到目标端；请按实际配置的保留时间规划容量 | + +Operator 会根据 PlatformLogForward 中的配置下发 TTL，以及 ClickHouse 的分片和副本数，因此准备好的集群必须能够满足这些值。chart 不包含目标端节点规格：ClickHouse 请按容量规划的档位选择，OpenSearch 请按 OpenSearch 的部署规格并使用相同的数据量和吞吐输入进行规划。 + +### 与 PlatformLogForward 对齐目标端配置 + +请以环境中实际部署的 CR 为准读取目标端配置，不要以本文档中的值为准： + +| 目标端 | 从哪里读取实际部署的配置 | 需要对齐什么 | +| --- | --- | --- | +| ClickHouse | `ClickHouseInstallation` CR：`spec.configuration.clusters[].layout.shardsCount` 和 `replicasCount` | 将 PlatformLogForward 的 `spec.externalStorage.shards` 和 `replicas` 设置为 CHI CR 中声明的值，并将 Secret 中的 `cluster` 设置为 `ON CLUSTER` 使用的集群名。数据链路会按这两个值展开，配置不一致会写入错误的 replica set | +| Kafka | `Kafka` CR（`spec.kafka.config`）、`KafkaNodePool`（旧布局为 `Kafka.spec.kafka`）中的 broker 数量和存储、`KafkaTopic` CR、`KafkaUser` CR | Secret 中的 topic 名（`topics.log`、`topics.event`、`topics.audit`）必须与 `KafkaTopic.spec.topicName` 以及 `KafkaUser` ACL 中的 topic 名完全一致，`kafkaClusterName` 必须与 Kafka CR 名称一致。分区数、副本因子和保留时间在 `KafkaTopic` CR 中设置（或使用你的 Kafka 部署中对应的 topic 配置）；副本因子不能超过 broker 数量。Broker 和 topic 的最大消息大小必须能够容纳审计批次——Kafka 默认的 1 MiB 不够，参考 CR 使用 10 MiB。保持 `auto.create.topics.enable` 为关闭，避免自动创建名称错误的 topic | +| OpenSearch | 实际部署的 OpenSearch 集群（其 operator CR 或运行它的 manifest），以及已生效的索引模板（`GET /_index_template`） | 索引的分片和副本不由 PlatformLogForward 控制。平台会下发低优先级模板，默认为 1 分片、1 副本；如果生产环境的节点数或 HA 策略需要不同的值，请在切换前应用更高优先级的 composable template，并通过 `GET /_index_template` 确认结果 | + +## 升级前检查清单 + +| 检查项 | 期望结果 | +| --- | --- | +| 源端 | 旧 ES、Kafka、ZooKeeper、lanaya 和 Razor 均在运行；`logcenter` 为 `Running` | +| 目标存储 | OpenSearch 3.7.0 或受支持的 ClickHouse 拓扑可访问；数据库/Keeper 要求已满足 | +| 目标 Kafka | 新的 bootstrap 可访问；topic 和日志用户权限已就绪 | +| 连接 Secret | 必需 key 齐全且有效；未复用旧 Secret | +| 访问与变更窗口 | 平台管理员可访问全局和业务集群；支持路径和变更窗口已确认 | +| 历史迁移 | 已取得本版本的迁移镜像；目标账号可以创建并写入所需的 schema 和数据 | + +## 升级流程概览 + +请按以下顺序执行。每步的卡点通过后才能进入下一步。 + +| 步骤 | 执行位置 | 操作 | 继续执行的条件 | +| --- | --- | --- | --- | +| 1 | 业务集群 | 准备目标存储、Kafka 和连接 Secret | 目标存储和 topic 可访问 | +| 2 | 业务集群 | 创建 `PlatformLogForward` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | +| 3 | 业务集群 | 如需历史数据，创建 `LegacyESMigration` | `PrecaptureReady`（已有迁移时为 `Succeeded`） | +| 4 | 全局与业务集群 | 执行旧插件的受控卸载 | 60 秒内 `ModuleInfo`、`ClusterPluginInstance`、`AppRelease` 均未出现，且新日志查询正常 | +| 5 | 业务集群 | 观察同一个 `LegacyESMigration` | `Phase=Succeeded` 且目标端查询通过 | +| 6 | 业务集群 | 保留受保护的源端存储卷 | 清理前需获得明确批准 | + +## 升级步骤 + +### 步骤 1：准备目标存储、Kafka 和连接 Secret + +**在业务集群执行。** + +在业务集群的 `cpaas-system` 中创建两个 Secret：一个用于目标存储，一个用于新的 Kafka。请勿覆盖或复用旧 Elasticsearch 和 Kafka 的 Secret。 + +#### 目标 OpenSearch + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 + namespace: cpaas-system +type: Opaque +stringData: + endpoints: "https://:9200" # 必填；逗号分隔的 HTTP(S) 地址；请把高可用协调节点或负载均衡地址放在最前面 + username: "" # 目标端允许匿名访问时可省略 + password: "" # 目标端允许匿名访问时可省略 + tls.ca: |- # 迁移历史数据且目标端使用 HTTPS 私有 CA 时必填，供迁移程序写入前校验目标端 + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +--- +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn # Kafka 连接 Secret 名称，创建 PlatformLogForward 时引用 + namespace: cpaas-system +type: Opaque +stringData: + bootstrap: "" # 必填；Kafka 地址，host:port 形式，逗号分隔 + kafkaClusterName: "" # 必填；Kafka broker 资源名称，必须与实际名称一致 + username: "" # 必填；Kafka 用户名 + password: "" # 必填；在 Alauda OS 节点或其他启用 FIPS 的主机上至少 32 个字符 + sasl_mechanism: "SCRAM-SHA-512" # 可选，默认 SCRAM-SHA-512 + topics.log: "ALAUDA_LOG_TOPIC" # 可选，日志 topic 名，默认 ALAUDA_LOG_TOPIC + topics.event: "ALAUDA_EVENT_TOPIC" # 可选，事件 topic 名，默认 ALAUDA_EVENT_TOPIC + topics.audit: "ALAUDA_AUDIT_TOPIC" # 可选，审计 topic 名，默认 ALAUDA_AUDIT_TOPIC + tls.ca: |- # Kafka 使用 TLS 且证书不被系统信任时必填 + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +`endpoints` 支持多个逗号分隔的 HTTP(S) 地址。部分数据链路只会使用第一个地址，因此请把高可用的负载均衡或协调节点地址放在最前面，不要放单个数据节点，并确保第一个 URL 前没有多余空格。 + +#### 目标 ClickHouse + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 + namespace: cpaas-system +type: Opaque +stringData: + endpoint: "https://:8443" # 必填；ClickHouse 地址，包含协议和端口 + cluster: "replicated" # 必须与 ClickHouse 集群名一致；ACP 基线默认为 replicated + database: "observability" # 目标数据库名，默认 observability + username: "" # ClickHouse 用户名 + password: "" # ClickHouse 密码 + tls.ca: |- # 目标端使用 HTTPS 私有 CA 时必填 + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +新的 Kafka 服务复用 OpenSearch 章节中的同一个 `platform-default-mq-conn`。 + +迁移历史数据时，请勿在目标存储连接 Secret 中设置 `tls.insecure_skip_verify: "true"`；请提供 `tls.ca`，以便迁移程序校验目标端。 + +### 步骤 2：创建新的数据链路 + +**在业务集群执行。** + +为你的目标端创建一个 `PlatformLogForward`。必须使用 `installMode: Fresh` 和 `log.alauda.io/legacy-es-upgrade: "true"` 注解，请勿使用 `installMode: Adopt`。 + +#### 目标 OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 固定的集群单例名称，请勿修改 + annotations: + log.alauda.io/legacy-es-upgrade: "true" # 固定值，进入旧 Elasticsearch 升级流程 +spec: + installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + externalStorage: + type: opensearch # 目标存储类型 + secretRef: + name: platform-default-os-conn # 步骤 1 创建的目标存储连接 Secret + namespace: cpaas-system + externalMessageQueue: + type: kafka # 消息队列类型，目前仅支持 kafka + secretRef: + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret + namespace: cpaas-system +``` + +#### 目标 ClickHouse + +ClickHouse 目标端必须设置 `output.type` 字段。 + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 固定的集群单例名称，请勿修改 + annotations: + log.alauda.io/legacy-es-upgrade: "true" # 固定值，进入旧 Elasticsearch 升级流程 +spec: + installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + output: + type: clickhouse # 目标为 ClickHouse 时必填 + externalStorage: + type: clickhouse # 目标存储类型 + shards: 1 # 目标 ClickHouse 的实际分片数 + replicas: 1 # 目标 ClickHouse 的实际副本数 + secretRef: + name: platform-default-ch-conn # 步骤 1 创建的目标存储连接 Secret + namespace: cpaas-system + externalMessageQueue: + type: kafka # 消息队列类型，目前仅支持 kafka + secretRef: + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret + namespace: cpaas-system +``` + +`externalStorage.shards` 和 `externalStorage.replicas` 必须与实际的 ClickHouse 拓扑一致。两者默认都是 `1`；在多分片或复制部署中填错会导致目标端拓扑只有一部分被使用。 + +`PlatformLogForward` 是集群级资源，请勿添加 `metadata.namespace`；`secretRef` 中的 `namespace` 字段仍然用于指定 `cpaas-system` 中的连接 Secret。CRD 默认值为 `aggregateVector.replicas: 3` 和 `razor.replicas: 2`；如果容量或调度规划需要不同的副本数，请显式设置。 + +将 YAML 保存为 `platform-log-forward.yaml` 并执行： + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +新数据链路不会立即就绪。平台会先创建链路，然后切换日志入口，最后等待旧集群中排队的数据被消费完；耗时取决于积压量。请观察状态直到完成，按 `Ctrl+C` 结束： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 列会从 `Provisioning` 变为 `Ready`，同时 `Ready` 列变为 `True`。只有看到 `Ready` 后才能继续。 + +如需跟踪进度或排查问题，可以查看状态 conditions： + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +请关注 `LegacyESUpgrade` 这一行：当 reason 变为 `LegacyESUpgradeCompleted` 时，说明日志入口已切换到新数据链路，且旧集群中排队的数据已消费完，本步骤完成。如果 reason 为 `Blocked`，`message` 会说明原因。 + +本步骤完成后，请生成或找到新的日志、事件和审计记录，确认可以从新目标端查询到，然后再继续。`LegacyESUpgrade` 未完成前，请勿卸载旧插件。 + +如果源端集群使用旧 Kafka，请勿停止或缩容 Kafka、ZooKeeper 或 lanaya。平台会自动通过旧链路排空排队数据，并在旧消费组 lag 归零时记录 `LegacyKafkaDrained`；`LegacyESUpgradeCompleted` 是本流程的卡点。 + +### 步骤 3：准备历史数据迁移（可选） + +**在业务集群执行。** + +只有在升级方案确认不需要历史 Elasticsearch 数据时才能跳过本步骤，并请记录该决定。请勿为了跳过迁移而删除源端 PVC 或 PV。 + +如果需要历史数据，请在卸载旧插件**之前**创建 `LegacyESMigration`，这样平台可以自动捕获最终的源端状态并解析源端存储卷。卸载后再创建迁移需要显式指定源端存储卷，且无法使用最终捕获。 + +#### 目标 OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # 固定的迁移名称，后续命令使用 + namespace: cpaas-system +spec: + image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest + source: + indexScope: # 必填，选择要迁移的历史索引 + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # 必须与 PlatformLogForward 的目标一致 + secretRef: + name: platform-default-os-conn # 与 PlatformLogForward 使用同一个连接 Secret + namespace: cpaas-system + options: + batchSize: 250 # 每批写入的文档数，1~100000，默认 250 + syncIntervalSeconds: 5 # 每批之间的最小等待时间（秒） + maxConcurrentJobs: 1 # 并发任务数，OpenSearch 最多 2 +``` + +#### 目标 ClickHouse + +将 `target.type` 改为 `clickhouse`，并把 Secret 指向 `platform-default-ch-conn`。 + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # 固定的迁移名称，后续命令使用 + namespace: cpaas-system +spec: + image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest + source: + indexScope: # 必填，选择要迁移的历史索引 + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # 必须与 PlatformLogForward 的目标一致 + secretRef: + name: platform-default-ch-conn # 与 PlatformLogForward 使用同一个连接 Secret + namespace: cpaas-system + options: + batchSize: 250 # 每批写入的文档数，1~100000，默认 250 + syncIntervalSeconds: 5 # 每批之间的最小等待时间（秒） + maxConcurrentJobs: 1 # 并发任务数，ClickHouse 只能为 1 +``` + +标准的 `indexScope` 分类如下： + +| `indexScope` 值 | 数据 | 完整索引名示例 | +| --- | --- | --- | +| `log-workload-*` | 应用和容器日志 | `log-workload-20260825` | +| `log-platform-*` | 平台组件日志 | `log-platform-20260825` | +| `log-system-*` | 系统日志 | `log-system-20260825` | +| `log-kubernetes-*` | Kubernetes 日志 | `log-kubernetes-20260825` | +| `event-*` | Kubernetes 事件 | `event-20260825` | +| `audit-*` | 审计日志 | `audit-20260825` | + +如果源端还包含项目级日志或计量数据，请将 `log-project-*` 或 `meter-*` 加入 `indexScope`，否则这些数据不会被迁移。除非确实只想迁移某一天的数据，否则不要把条目缩小到单日索引（如 `audit-20260825`）。`source` 和 `target` 在创建后无法修改。上面的迁移参数是默认值，仅在迁移方案需要时调整。 + +在推荐流程中不要设置 `source.pvcRefs`：只要旧 Elasticsearch StatefulSet 仍存在，平台会自动发现源端存储卷，并记录在 `status.resolvedPvcRefs` 中。如果迁移报告 `SourcePVCsUnavailable`，请停止并联系支持；请勿删除或修改迁移资源。 + +应用该资源并等待 `PrecaptureReady`： + +```bash +kubectl -n cpaas-system apply -f legacy-es-migration.yaml + +# 观察 phase；按 Ctrl+C 结束 +kubectl -n cpaas-system get legacyesmigration platform-es-history -w + +# phase 不推进时查看 conditions +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +只有在旧插件卸载之后才会开始复制数据，因此在此之前 phase 会一直停留在 `PrecaptureReady`。只有 phase 为 `PrecaptureReady`（已完成的迁移为 `Succeeded`）时才继续步骤 4。如果 phase 为 `Blocked`，请勿删除或重建迁移资源；请阅读 condition message 并联系支持。 + +:::info +迁移过程中新数据链路会持续接收日志、事件和审计数据。请保持新数据链路及其目标连接不变。如果迁移无法完成，请停止并联系支持。 +::: + +请使用本 ACP 4.4 日志版本提供的迁移镜像。如果目标数据已经完整但迁移一直停留在 `Running`，请停止并联系支持；请勿删除迁移资源或源端存储卷。 + +### 步骤 4：卸载旧 Elasticsearch 存储插件 + +:::warning +只有在以下条件全部满足时才能执行本步骤： + +- `PlatformLogForward/platform-default` 为 `Ready`，且其 `LegacyESUpgrade` condition 的 reason 为 `LegacyESUpgradeCompleted`。 +- 如需历史迁移，`LegacyESMigration/platform-es-history` 为 `PrecaptureReady` 或 `Succeeded`。 +- 平台已报告旧 Elasticsearch 存储卷（PVC/PV）的保护完成。请勿修改或移除该保护；如果平台报告保护失败，请停止并联系支持。 +- 如果源端使用旧 Kafka，平台已确认所有排队数据均已排空。请勿通过停止或缩容 Kafka、ZooKeeper、lanaya 或 Elasticsearch 来强制达到该状态。 +::: + +:::warning +本流程会临时清理 `logcenter` 的平台发现字段，以便卸载旧插件。请仅在批准的变更窗口内执行，并请勿修改或删除 logagent 及其依赖。 +::: + +如果平台提供了本次升级支持的插件卸载操作，请先按平台或支持人员的说明执行。如果该操作被拒绝，或平台团队要求执行受控流程，请使用下面的步骤。 + +两次删除必须连续执行：中间不要等待 `ModuleInfo` 消失、不要检查 `AppRelease`，也不要执行其他检查。 + +**在全局集群：开始关键步骤** + +将 `CLUSTER` 设置为业务集群在全局集群中注册的名称，然后执行： + +```bash +set -euo pipefail + +CLUSTER= + +# 1. 解析目标 ModuleInfo。结果不唯一时停止。 +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" +if [ "$MODULE_COUNT" -gt 1 ]; then + echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 + exit 1 +fi + +if [ "$MODULE_COUNT" -eq 1 ]; then + MODULE_INFO="$MODULE_INFOS" + MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" + MODULE_CONFIG="logcenter-${MODULE_VERSION}" + + # 2. 修改前先备份当前的管理对象。 + kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml + kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml + kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml + + # 3. 清理发现标记，这一步同时绕过 logagent 的依赖检查。 + kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + + # 4. 删除请求被接受后立即返回，不要在此等待。 + kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false +else + echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." +fi +``` + +**在业务集群：完成关键步骤** + +立即把 kubectl context 切换到业务集群，不要先执行任何等待或校验。执行： + +```bash +# 5. 删除单集群安装记录，避免其重新创建 ModuleInfo。 +kubectl delete clusterplugininstance logcenter --ignore-not-found +``` + +**等待旧数据链路移除** + +切回全局集群，等待 `ModuleInfo` 消失： + +```bash +CLUSTER= +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" + +if [ -n "$MODULE_INFO" ]; then + kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No ModuleInfo remains for cluster $CLUSTER." +fi +``` + +切换到业务集群，等待旧 `AppRelease` 被删除： + +```bash +kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +完成这些步骤后，等待 60 秒并确认这些资源没有重新出现。每条命令都不应返回资源。 + +**在全局集群** + +```bash +kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found +``` + +**在业务集群** + +```bash +kubectl get clusterplugininstance logcenter --ignore-not-found +kubectl -n cpaas-system get apprelease logcenter --ignore-not-found +kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found +``` + +如果 `ClusterPluginInstance/logcenter` 仍然存在或重新出现，请再次删除并重复检查；只要它存在，平台就可能重新创建 `ModuleInfo`。如果 `ModuleInfo` 也重新出现，请先删除 `ClusterPluginInstance/logcenter`，再删除新的 `ModuleInfo`。请勿手动恢复被清理的发现字段；只需确认 `ModuleInfo`、`ClusterPluginInstance` 和 `AppRelease` 不再出现。如果全局集群的步骤在删除 `ModuleInfo` 之前失败，请勿执行本次的业务集群步骤：先修复原因，再重新执行全局集群步骤。 + +稳定性检查通过后，生成新的日志、事件和审计记录，并确认可以从新目标端查询到。如果新数据链路不健康，请勿继续迁移校验。 + +### 步骤 5：完成并校验历史迁移（可选） + +**在业务集群执行。** + +如果已创建 `LegacyESMigration`，请持续观察同一个资源。请勿删除或重建它。phase 应从 `PrecaptureReady` 依次变为 `Validating`、`Running`（或 `Retrying`），最终变为 `Succeeded`。 + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history -w + +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' +``` + +**校验：** + +- `status.phase` 为 `Succeeded`。 +- `status.tasks` 中的每一项都成功。 +- 针对新目标端的抽样查询能返回所选每个 `indexScope` 分类的预期历史数据。 +- 受保护的源端 PVC 和 PV 仍然存在且处于绑定状态。 + +如果 phase 为 `Blocked` 或 `Failed`，请勿删除迁移资源、Job 或源端存储卷；请联系支持。 + +如果未创建迁移，请记录「不需要历史数据」的明确决定，并在该决定获批前保留受保护的源端存储卷。 + +### 步骤 6：保留的存储卷 + +受保护的旧 Elasticsearch PVC 和 PV 会被保留。请在迁移和目标端校验完成前保留它们。请勿移除保护注解、删除 PVC/PV 或移除 finalizer。校验完成后如需释放这些存储卷，请联系 Alauda 支持或按照单独的、已批准的清理流程执行。 + +本次升级只保留 ES 的 PVC 和 PV。旧 Kafka 和 ZooKeeper 的存储卷不在保留范围内。 + +请导出或记录最终的 PLF conditions、迁移状态、目标端查询结果以及保留的 PVC/PV 名称，用于实施交接。请勿在本次升级中移除保护注解或删除这些存储卷。 + +## 完成检查清单 + +| 状态 | 期望 | +| --- | --- | +| `PlatformLogForward/platform-default` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | +| `LegacyESMigration/platform-es-history`（如已创建） | `Phase=Succeeded` | +| 新日志查询 | 新的日志、事件和审计记录可以通过新查询链路返回 | +| 旧插件卸载 | `ModuleInfo`、`ClusterPluginInstance/logcenter`、`AppRelease/logcenter` 均不存在，旧 ES 工作负载已消失 | +| 源端存储卷 | 受保护的旧 ES PVC 和 PV 仍然存在 | + +如果任何状态不符合预期，请停止并联系 Alauda 支持。请勿通过删除迁移资源、源端存储卷或目标端数据来绕过故障。 + +## 步骤受阻时 + +- 平台以 `moduleinfo is depended by ...` 拒绝卸载插件：请停止，不要修改 logagent，重新执行受控流程；如果仍然失败，请联系支持。 +- 迁移为 `Blocked`、`Failed` 或一直停留在 `Running`：请勿删除迁移资源、Job、源端存储卷或目标端数据；请联系支持。 +- `ModuleInfo` 或 `ClusterPluginInstance` 重新出现：请先删除 `ClusterPluginInstance`，再删除新的 `ModuleInfo`，并重复 60 秒稳定性检查。 +- `AppRelease/logcenter` 未消失：请勿移除 finalizer；请联系支持。 +- 卸载后新日志查询失败：请停止并联系支持。请勿删除保留的存储卷。 From 3eb0b661a0397536f2bf940f0b4cdee7f52ebeeb Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 08:52:36 +0800 Subject: [PATCH 13/38] Revert "docs: add Chinese translation for the ES upgrade guide" This reverts commit 9f554484135182a4326ad6cb0b925d923d4f824d. --- docs/zh/upgrade/index.mdx | 553 -------------------------------------- 1 file changed, 553 deletions(-) delete mode 100644 docs/zh/upgrade/index.mdx diff --git a/docs/zh/upgrade/index.mdx b/docs/zh/upgrade/index.mdx deleted file mode 100644 index ad4af65..0000000 --- a/docs/zh/upgrade/index.mdx +++ /dev/null @@ -1,553 +0,0 @@ ---- -weight: 15 -sourceSHA: ae9e0e3a793eebf71d8cd10c294b316313c8dc578a27a9a099cf1b94dee84ae7 ---- - -# 升级 - -本文介绍如何升级现有 ACP 部署中的日志组件。 - -:::info -如果集群已使用 ClickHouse 或 OpenSearch 存储日志，则无需执行本升级指南，请参考[安装](https://docs.alauda.cn/logging-service/4.4/install_log.html)完成插件安装或更新。 -::: - -## 简介 - -本指南用于将使用 **Alauda Container Platform Log Storage for Elasticsearch** 存储日志的集群升级到 ACP 4.4。升级后，新的日志数据将写入 ClickHouse 或 OpenSearch 3.7.0 集群。 - -升级过程中 Elasticsearch 及其存储卷保持不动，平台会在旧链路旁边创建新链路，因此日志采集不会中断： - -1. 平台创建新的日志接收与存储链路，并开始将新的日志、事件和审计数据写入该链路。 -2. 旧 Elasticsearch 集群和旧链路保持运行，直到队列中的数据消费完成。 -3. 切换和排空完成后，PlatformLogForward 会报告 `LegacyESUpgradeCompleted`。 -4. 如果需要保留历史数据，请创建迁移资源，并在卸载旧插件前等待其进入 `PrecaptureReady`。 -5. 确认升级程序已完成旧 Elasticsearch 存储卷保护，然后卸载旧插件。 -6. 持续观察同一个迁移资源直到其成功，并在校验完成前保留受保护的源端存储卷。 - -:::warning -在本指南明确要求之前，请勿卸载 Elasticsearch 存储插件、停止旧数据链路，或删除其 PVC 和 PV。提前执行这些操作可能导致历史数据不可用，或导致无法捕获源端元数据快照。 -::: - -## 适用场景 - -当集群符合下面的源端和目标端条件时，请使用本指南。 - -| 项目 | 值 | -| --- | --- | -| 源端 | 已安装并运行 **Alauda Container Platform Log Storage for Elasticsearch** 的业务集群 | -| 目标平台 | ACP 4.4 | -| 目标存储 | 单独准备的 ClickHouse 或 OpenSearch 3.7.0 集群 | -| 支持的源端日志插件版本 | 4.2.x、4.3.x | - -全新安装，或集群已使用 ClickHouse / OpenSearch 存储日志时，请勿使用本指南，请参考[安装](https://docs.alauda.cn/logging-service/4.4/install_log.html)。 - -## 适用人员 - -本文是面向 Alauda 实施工程师或平台管理员的实施手册，不是最终用户的自助操作流程。 - -- 请使用可同时访问全局管理集群和目标业务集群的平台账号。 -- 尽量为每个集群使用独立的终端或 kubectl context。本文每个命令块都说明了在哪个集群执行。 -- 执行每个命令块前，请先用 `kubectl config current-context` 确认当前 context。 -- 在托管环境中，开始操作前请与 Alauda 支持确认变更窗口和升级路径。 - -## 前置条件 - -开始前，请确保： - -1. ACP 4.4 平台升级已完成，且日志组件可以升级。请在本流程中升级日志控制组件，并保持 **Alauda Container Platform Log Storage for Elasticsearch** 处于安装状态。 -2. 已从 **Alauda Cloud** 下载 ACP 4.4 日志插件包，且该插件包已上传到集群的插件市场。 -3. 已单独准备本次升级所需的目标存储和消息队列。本次升级不会复用 Elasticsearch 插件自带的存储和 Kafka，需要准备： - - 对于 OpenSearch 3.7.0，`analysis-ik` 插件可选。未安装时，日志查询使用 OpenSearch 内置的 standard 分词器：查询仍可正常执行，但中文全文检索效果会下降。如果现场需要中文检索质量，请在每个节点安装与 3.7.0 匹配的该插件。连接账号需要具备索引模板和生命周期策略的管理权限，以及日志数据的读写权限。 - - 对于 ClickHouse，请使用支持 `ReplicatedMergeTree` 和 Keeper/ZooKeeper 的复制集群。连接 Secret 中的 `cluster` 值必须与 ClickHouse 集群名一致；目标数据库必须在创建 PlatformLogForward 之前存在，如果新链路因 `UNKNOWN_DATABASE` 一直未就绪，请先创建数据库并等待下一次调谐。连接账号需要具备建表和改表、数据读写权限，以及执行 `SYSTEM DROP DNS CACHE` 的权限。 - - 新的 Kafka 服务，并已创建 `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC` 和 `ALAUDA_AUDIT_TOPIC` 三个 topic，日志 Kafka 用户已获得这些 topic 及相关消费组的访问权限。 - - 按各自产品文档准备好的目标存储和 Kafka 连接信息。 -4. 如果需要迁移历史数据，已取得本版本提供的迁移镜像，并包含完整的 registry、tag 或 digest。请勿复用旧版本的镜像。 -5. 已获得批准的变更窗口，且平台管理员可同时访问全局管理集群和目标业务集群。在托管环境中，请与 Alauda 支持协同。 -6. 旧 Elasticsearch、Kafka、ZooKeeper、lanaya 和 Razor 工作负载仍在运行。在本指南要求之前，请勿停止、缩容或删除它们。 - -:::warning -升级期间，请勿停止、缩容或删除旧 Elasticsearch、Kafka、ZooKeeper、lanaya 或 Razor 工作负载；请勿删除 PVC、PV 或保护 finalizer；请勿删除或重建 `LegacyESMigration` 资源。这些操作可能导致历史数据不可用，或破坏迁移边界。 -::: - -## 目标存储准备 - -Operator 不会创建外部 OpenSearch/ClickHouse 集群和新的 Kafka 服务，需要单独创建，并通过步骤 1 中的 Secret 提供连接信息。 - -请参考[日志组件容量规划](https://docs.alauda.cn/logging-service/4.4/architecture/capacity_planning.html)规划容量，且不要低于当前 logcenter 部署的规格。目标存储需要能够容纳迁移的历史数据和新产生的数据。该指南中的参考磁盘配置为 `6000 IOPS`、`250 MB/s` 读写、独立 SSD 挂载；如果实际存储性能低于该配置，请选择更大规格。 - -下表中的旧 logcenter 参数是当前部署的基线，而不是目标端配置。它们来自 `chart-alauda-log-center` 的默认值，不同现场可能不同，规划前请先核对实际部署值： - -| 旧 logcenter chart 参数 | 典型基线 | 需要为目标端准备什么 | -| --- | --- | --- | -| `elasticsearch.storage.node_size`、`node_replicas` | 每个数据节点 200 Gi；`node_replicas` 表示数据节点数量（默认 1，小规模档位为 3） | 按历史数据量、新增数据量和目标端 HA 策略规划目标存储 | -| `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | 目标端使用独立存储，不复用旧的本地路径 | -| `logging.esReplicas`、`logging.shards` | 索引级配置：1 副本、按类型设置分片数 | 仅适用于 OpenSearch：作为索引模板的基线；如果目标端需要不同的值，请覆盖模板。它们不适用于 ClickHouse——ClickHouse 使用 `externalStorage.shards` 和 `replicas` 描述集群拓扑（见下方对齐表） | -| `kafka.retention_hours` | 48 小时 | 在目标 `KafkaTopic` CR 中设置相同的保留时间；Kafka broker 的容量请按你的 Kafka 部署规划，不要沿用该旧 chart 参数 | -| `logging.ttl` | 日志 7 天；事件/审计 180 天；计量 540 天 | PlatformLogForward 会把相同的 TTL 下发到目标端；请按实际配置的保留时间规划容量 | - -Operator 会根据 PlatformLogForward 中的配置下发 TTL，以及 ClickHouse 的分片和副本数，因此准备好的集群必须能够满足这些值。chart 不包含目标端节点规格：ClickHouse 请按容量规划的档位选择，OpenSearch 请按 OpenSearch 的部署规格并使用相同的数据量和吞吐输入进行规划。 - -### 与 PlatformLogForward 对齐目标端配置 - -请以环境中实际部署的 CR 为准读取目标端配置，不要以本文档中的值为准： - -| 目标端 | 从哪里读取实际部署的配置 | 需要对齐什么 | -| --- | --- | --- | -| ClickHouse | `ClickHouseInstallation` CR：`spec.configuration.clusters[].layout.shardsCount` 和 `replicasCount` | 将 PlatformLogForward 的 `spec.externalStorage.shards` 和 `replicas` 设置为 CHI CR 中声明的值，并将 Secret 中的 `cluster` 设置为 `ON CLUSTER` 使用的集群名。数据链路会按这两个值展开，配置不一致会写入错误的 replica set | -| Kafka | `Kafka` CR（`spec.kafka.config`）、`KafkaNodePool`（旧布局为 `Kafka.spec.kafka`）中的 broker 数量和存储、`KafkaTopic` CR、`KafkaUser` CR | Secret 中的 topic 名（`topics.log`、`topics.event`、`topics.audit`）必须与 `KafkaTopic.spec.topicName` 以及 `KafkaUser` ACL 中的 topic 名完全一致，`kafkaClusterName` 必须与 Kafka CR 名称一致。分区数、副本因子和保留时间在 `KafkaTopic` CR 中设置（或使用你的 Kafka 部署中对应的 topic 配置）；副本因子不能超过 broker 数量。Broker 和 topic 的最大消息大小必须能够容纳审计批次——Kafka 默认的 1 MiB 不够，参考 CR 使用 10 MiB。保持 `auto.create.topics.enable` 为关闭，避免自动创建名称错误的 topic | -| OpenSearch | 实际部署的 OpenSearch 集群（其 operator CR 或运行它的 manifest），以及已生效的索引模板（`GET /_index_template`） | 索引的分片和副本不由 PlatformLogForward 控制。平台会下发低优先级模板，默认为 1 分片、1 副本；如果生产环境的节点数或 HA 策略需要不同的值，请在切换前应用更高优先级的 composable template，并通过 `GET /_index_template` 确认结果 | - -## 升级前检查清单 - -| 检查项 | 期望结果 | -| --- | --- | -| 源端 | 旧 ES、Kafka、ZooKeeper、lanaya 和 Razor 均在运行；`logcenter` 为 `Running` | -| 目标存储 | OpenSearch 3.7.0 或受支持的 ClickHouse 拓扑可访问；数据库/Keeper 要求已满足 | -| 目标 Kafka | 新的 bootstrap 可访问；topic 和日志用户权限已就绪 | -| 连接 Secret | 必需 key 齐全且有效；未复用旧 Secret | -| 访问与变更窗口 | 平台管理员可访问全局和业务集群；支持路径和变更窗口已确认 | -| 历史迁移 | 已取得本版本的迁移镜像；目标账号可以创建并写入所需的 schema 和数据 | - -## 升级流程概览 - -请按以下顺序执行。每步的卡点通过后才能进入下一步。 - -| 步骤 | 执行位置 | 操作 | 继续执行的条件 | -| --- | --- | --- | --- | -| 1 | 业务集群 | 准备目标存储、Kafka 和连接 Secret | 目标存储和 topic 可访问 | -| 2 | 业务集群 | 创建 `PlatformLogForward` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | -| 3 | 业务集群 | 如需历史数据，创建 `LegacyESMigration` | `PrecaptureReady`（已有迁移时为 `Succeeded`） | -| 4 | 全局与业务集群 | 执行旧插件的受控卸载 | 60 秒内 `ModuleInfo`、`ClusterPluginInstance`、`AppRelease` 均未出现，且新日志查询正常 | -| 5 | 业务集群 | 观察同一个 `LegacyESMigration` | `Phase=Succeeded` 且目标端查询通过 | -| 6 | 业务集群 | 保留受保护的源端存储卷 | 清理前需获得明确批准 | - -## 升级步骤 - -### 步骤 1：准备目标存储、Kafka 和连接 Secret - -**在业务集群执行。** - -在业务集群的 `cpaas-system` 中创建两个 Secret：一个用于目标存储，一个用于新的 Kafka。请勿覆盖或复用旧 Elasticsearch 和 Kafka 的 Secret。 - -#### 目标 OpenSearch - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-os-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 - namespace: cpaas-system -type: Opaque -stringData: - endpoints: "https://:9200" # 必填；逗号分隔的 HTTP(S) 地址；请把高可用协调节点或负载均衡地址放在最前面 - username: "" # 目标端允许匿名访问时可省略 - password: "" # 目标端允许匿名访问时可省略 - tls.ca: |- # 迁移历史数据且目标端使用 HTTPS 私有 CA 时必填，供迁移程序写入前校验目标端 - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- ---- -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-mq-conn # Kafka 连接 Secret 名称，创建 PlatformLogForward 时引用 - namespace: cpaas-system -type: Opaque -stringData: - bootstrap: "" # 必填；Kafka 地址，host:port 形式，逗号分隔 - kafkaClusterName: "" # 必填；Kafka broker 资源名称，必须与实际名称一致 - username: "" # 必填；Kafka 用户名 - password: "" # 必填；在 Alauda OS 节点或其他启用 FIPS 的主机上至少 32 个字符 - sasl_mechanism: "SCRAM-SHA-512" # 可选，默认 SCRAM-SHA-512 - topics.log: "ALAUDA_LOG_TOPIC" # 可选，日志 topic 名，默认 ALAUDA_LOG_TOPIC - topics.event: "ALAUDA_EVENT_TOPIC" # 可选，事件 topic 名，默认 ALAUDA_EVENT_TOPIC - topics.audit: "ALAUDA_AUDIT_TOPIC" # 可选，审计 topic 名，默认 ALAUDA_AUDIT_TOPIC - tls.ca: |- # Kafka 使用 TLS 且证书不被系统信任时必填 - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -`endpoints` 支持多个逗号分隔的 HTTP(S) 地址。部分数据链路只会使用第一个地址，因此请把高可用的负载均衡或协调节点地址放在最前面，不要放单个数据节点，并确保第一个 URL 前没有多余空格。 - -#### 目标 ClickHouse - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-ch-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 - namespace: cpaas-system -type: Opaque -stringData: - endpoint: "https://:8443" # 必填；ClickHouse 地址，包含协议和端口 - cluster: "replicated" # 必须与 ClickHouse 集群名一致；ACP 基线默认为 replicated - database: "observability" # 目标数据库名，默认 observability - username: "" # ClickHouse 用户名 - password: "" # ClickHouse 密码 - tls.ca: |- # 目标端使用 HTTPS 私有 CA 时必填 - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -新的 Kafka 服务复用 OpenSearch 章节中的同一个 `platform-default-mq-conn`。 - -迁移历史数据时，请勿在目标存储连接 Secret 中设置 `tls.insecure_skip_verify: "true"`；请提供 `tls.ca`，以便迁移程序校验目标端。 - -### 步骤 2：创建新的数据链路 - -**在业务集群执行。** - -为你的目标端创建一个 `PlatformLogForward`。必须使用 `installMode: Fresh` 和 `log.alauda.io/legacy-es-upgrade: "true"` 注解，请勿使用 `installMode: Adopt`。 - -#### 目标 OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # 固定的集群单例名称，请勿修改 - annotations: - log.alauda.io/legacy-es-upgrade: "true" # 固定值，进入旧 Elasticsearch 升级流程 -spec: - installMode: Fresh # 始终为 Fresh，请勿改为 Adopt - externalStorage: - type: opensearch # 目标存储类型 - secretRef: - name: platform-default-os-conn # 步骤 1 创建的目标存储连接 Secret - namespace: cpaas-system - externalMessageQueue: - type: kafka # 消息队列类型，目前仅支持 kafka - secretRef: - name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret - namespace: cpaas-system -``` - -#### 目标 ClickHouse - -ClickHouse 目标端必须设置 `output.type` 字段。 - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # 固定的集群单例名称，请勿修改 - annotations: - log.alauda.io/legacy-es-upgrade: "true" # 固定值，进入旧 Elasticsearch 升级流程 -spec: - installMode: Fresh # 始终为 Fresh，请勿改为 Adopt - output: - type: clickhouse # 目标为 ClickHouse 时必填 - externalStorage: - type: clickhouse # 目标存储类型 - shards: 1 # 目标 ClickHouse 的实际分片数 - replicas: 1 # 目标 ClickHouse 的实际副本数 - secretRef: - name: platform-default-ch-conn # 步骤 1 创建的目标存储连接 Secret - namespace: cpaas-system - externalMessageQueue: - type: kafka # 消息队列类型，目前仅支持 kafka - secretRef: - name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret - namespace: cpaas-system -``` - -`externalStorage.shards` 和 `externalStorage.replicas` 必须与实际的 ClickHouse 拓扑一致。两者默认都是 `1`；在多分片或复制部署中填错会导致目标端拓扑只有一部分被使用。 - -`PlatformLogForward` 是集群级资源，请勿添加 `metadata.namespace`；`secretRef` 中的 `namespace` 字段仍然用于指定 `cpaas-system` 中的连接 Secret。CRD 默认值为 `aggregateVector.replicas: 3` 和 `razor.replicas: 2`；如果容量或调度规划需要不同的副本数，请显式设置。 - -将 YAML 保存为 `platform-log-forward.yaml` 并执行： - -```bash -kubectl apply -f platform-log-forward.yaml -``` - -新数据链路不会立即就绪。平台会先创建链路，然后切换日志入口，最后等待旧集群中排队的数据被消费完；耗时取决于积压量。请观察状态直到完成，按 `Ctrl+C` 结束： - -```bash -kubectl get platformlogforward platform-default -w -``` - -`Phase` 列会从 `Provisioning` 变为 `Ready`，同时 `Ready` 列变为 `True`。只有看到 `Ready` 后才能继续。 - -如需跟踪进度或排查问题，可以查看状态 conditions： - -```bash -kubectl get platformlogforward platform-default \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' -``` - -请关注 `LegacyESUpgrade` 这一行：当 reason 变为 `LegacyESUpgradeCompleted` 时，说明日志入口已切换到新数据链路，且旧集群中排队的数据已消费完，本步骤完成。如果 reason 为 `Blocked`，`message` 会说明原因。 - -本步骤完成后，请生成或找到新的日志、事件和审计记录，确认可以从新目标端查询到，然后再继续。`LegacyESUpgrade` 未完成前，请勿卸载旧插件。 - -如果源端集群使用旧 Kafka，请勿停止或缩容 Kafka、ZooKeeper 或 lanaya。平台会自动通过旧链路排空排队数据，并在旧消费组 lag 归零时记录 `LegacyKafkaDrained`；`LegacyESUpgradeCompleted` 是本流程的卡点。 - -### 步骤 3：准备历史数据迁移（可选） - -**在业务集群执行。** - -只有在升级方案确认不需要历史 Elasticsearch 数据时才能跳过本步骤，并请记录该决定。请勿为了跳过迁移而删除源端 PVC 或 PV。 - -如果需要历史数据，请在卸载旧插件**之前**创建 `LegacyESMigration`，这样平台可以自动捕获最终的源端状态并解析源端存储卷。卸载后再创建迁移需要显式指定源端存储卷，且无法使用最终捕获。 - -#### 目标 OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # 固定的迁移名称，后续命令使用 - namespace: cpaas-system -spec: - image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest - source: - indexScope: # 必填，选择要迁移的历史索引 - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - target: - type: opensearch # 必须与 PlatformLogForward 的目标一致 - secretRef: - name: platform-default-os-conn # 与 PlatformLogForward 使用同一个连接 Secret - namespace: cpaas-system - options: - batchSize: 250 # 每批写入的文档数，1~100000，默认 250 - syncIntervalSeconds: 5 # 每批之间的最小等待时间（秒） - maxConcurrentJobs: 1 # 并发任务数，OpenSearch 最多 2 -``` - -#### 目标 ClickHouse - -将 `target.type` 改为 `clickhouse`，并把 Secret 指向 `platform-default-ch-conn`。 - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # 固定的迁移名称，后续命令使用 - namespace: cpaas-system -spec: - image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest - source: - indexScope: # 必填，选择要迁移的历史索引 - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - target: - type: clickhouse # 必须与 PlatformLogForward 的目标一致 - secretRef: - name: platform-default-ch-conn # 与 PlatformLogForward 使用同一个连接 Secret - namespace: cpaas-system - options: - batchSize: 250 # 每批写入的文档数，1~100000，默认 250 - syncIntervalSeconds: 5 # 每批之间的最小等待时间（秒） - maxConcurrentJobs: 1 # 并发任务数，ClickHouse 只能为 1 -``` - -标准的 `indexScope` 分类如下： - -| `indexScope` 值 | 数据 | 完整索引名示例 | -| --- | --- | --- | -| `log-workload-*` | 应用和容器日志 | `log-workload-20260825` | -| `log-platform-*` | 平台组件日志 | `log-platform-20260825` | -| `log-system-*` | 系统日志 | `log-system-20260825` | -| `log-kubernetes-*` | Kubernetes 日志 | `log-kubernetes-20260825` | -| `event-*` | Kubernetes 事件 | `event-20260825` | -| `audit-*` | 审计日志 | `audit-20260825` | - -如果源端还包含项目级日志或计量数据，请将 `log-project-*` 或 `meter-*` 加入 `indexScope`，否则这些数据不会被迁移。除非确实只想迁移某一天的数据，否则不要把条目缩小到单日索引（如 `audit-20260825`）。`source` 和 `target` 在创建后无法修改。上面的迁移参数是默认值，仅在迁移方案需要时调整。 - -在推荐流程中不要设置 `source.pvcRefs`：只要旧 Elasticsearch StatefulSet 仍存在，平台会自动发现源端存储卷，并记录在 `status.resolvedPvcRefs` 中。如果迁移报告 `SourcePVCsUnavailable`，请停止并联系支持；请勿删除或修改迁移资源。 - -应用该资源并等待 `PrecaptureReady`： - -```bash -kubectl -n cpaas-system apply -f legacy-es-migration.yaml - -# 观察 phase；按 Ctrl+C 结束 -kubectl -n cpaas-system get legacyesmigration platform-es-history -w - -# phase 不推进时查看 conditions -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' -``` - -只有在旧插件卸载之后才会开始复制数据，因此在此之前 phase 会一直停留在 `PrecaptureReady`。只有 phase 为 `PrecaptureReady`（已完成的迁移为 `Succeeded`）时才继续步骤 4。如果 phase 为 `Blocked`，请勿删除或重建迁移资源；请阅读 condition message 并联系支持。 - -:::info -迁移过程中新数据链路会持续接收日志、事件和审计数据。请保持新数据链路及其目标连接不变。如果迁移无法完成，请停止并联系支持。 -::: - -请使用本 ACP 4.4 日志版本提供的迁移镜像。如果目标数据已经完整但迁移一直停留在 `Running`，请停止并联系支持；请勿删除迁移资源或源端存储卷。 - -### 步骤 4：卸载旧 Elasticsearch 存储插件 - -:::warning -只有在以下条件全部满足时才能执行本步骤： - -- `PlatformLogForward/platform-default` 为 `Ready`，且其 `LegacyESUpgrade` condition 的 reason 为 `LegacyESUpgradeCompleted`。 -- 如需历史迁移，`LegacyESMigration/platform-es-history` 为 `PrecaptureReady` 或 `Succeeded`。 -- 平台已报告旧 Elasticsearch 存储卷（PVC/PV）的保护完成。请勿修改或移除该保护；如果平台报告保护失败，请停止并联系支持。 -- 如果源端使用旧 Kafka，平台已确认所有排队数据均已排空。请勿通过停止或缩容 Kafka、ZooKeeper、lanaya 或 Elasticsearch 来强制达到该状态。 -::: - -:::warning -本流程会临时清理 `logcenter` 的平台发现字段，以便卸载旧插件。请仅在批准的变更窗口内执行，并请勿修改或删除 logagent 及其依赖。 -::: - -如果平台提供了本次升级支持的插件卸载操作，请先按平台或支持人员的说明执行。如果该操作被拒绝，或平台团队要求执行受控流程，请使用下面的步骤。 - -两次删除必须连续执行：中间不要等待 `ModuleInfo` 消失、不要检查 `AppRelease`，也不要执行其他检查。 - -**在全局集群：开始关键步骤** - -将 `CLUSTER` 设置为业务集群在全局集群中注册的名称，然后执行： - -```bash -set -euo pipefail - -CLUSTER= - -# 1. 解析目标 ModuleInfo。结果不唯一时停止。 -MODULE_INFOS="$(kubectl get moduleplugin logcenter \ - -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" -MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" -if [ "$MODULE_COUNT" -gt 1 ]; then - echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 - exit 1 -fi - -if [ "$MODULE_COUNT" -eq 1 ]; then - MODULE_INFO="$MODULE_INFOS" - MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" - MODULE_CONFIG="logcenter-${MODULE_VERSION}" - - # 2. 修改前先备份当前的管理对象。 - kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml - kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml - kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml - - # 3. 清理发现标记，这一步同时绕过 logagent 的依赖检查。 - kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' - kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' - - # 4. 删除请求被接受后立即返回，不要在此等待。 - kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false -else - echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." -fi -``` - -**在业务集群：完成关键步骤** - -立即把 kubectl context 切换到业务集群，不要先执行任何等待或校验。执行： - -```bash -# 5. 删除单集群安装记录，避免其重新创建 ModuleInfo。 -kubectl delete clusterplugininstance logcenter --ignore-not-found -``` - -**等待旧数据链路移除** - -切回全局集群，等待 `ModuleInfo` 消失： - -```bash -CLUSTER= -MODULE_INFOS="$(kubectl get moduleplugin logcenter \ - -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" -MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" - -if [ -n "$MODULE_INFO" ]; then - kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m -else - echo "No ModuleInfo remains for cluster $CLUSTER." -fi -``` - -切换到业务集群，等待旧 `AppRelease` 被删除： - -```bash -kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m -``` - -完成这些步骤后，等待 60 秒并确认这些资源没有重新出现。每条命令都不应返回资源。 - -**在全局集群** - -```bash -kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found -``` - -**在业务集群** - -```bash -kubectl get clusterplugininstance logcenter --ignore-not-found -kubectl -n cpaas-system get apprelease logcenter --ignore-not-found -kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found -``` - -如果 `ClusterPluginInstance/logcenter` 仍然存在或重新出现，请再次删除并重复检查；只要它存在，平台就可能重新创建 `ModuleInfo`。如果 `ModuleInfo` 也重新出现，请先删除 `ClusterPluginInstance/logcenter`，再删除新的 `ModuleInfo`。请勿手动恢复被清理的发现字段；只需确认 `ModuleInfo`、`ClusterPluginInstance` 和 `AppRelease` 不再出现。如果全局集群的步骤在删除 `ModuleInfo` 之前失败，请勿执行本次的业务集群步骤：先修复原因，再重新执行全局集群步骤。 - -稳定性检查通过后，生成新的日志、事件和审计记录，并确认可以从新目标端查询到。如果新数据链路不健康，请勿继续迁移校验。 - -### 步骤 5：完成并校验历史迁移（可选） - -**在业务集群执行。** - -如果已创建 `LegacyESMigration`，请持续观察同一个资源。请勿删除或重建它。phase 应从 `PrecaptureReady` 依次变为 `Validating`、`Running`（或 `Retrying`），最终变为 `Succeeded`。 - -```bash -kubectl -n cpaas-system get legacyesmigration platform-es-history -w - -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' -``` - -**校验：** - -- `status.phase` 为 `Succeeded`。 -- `status.tasks` 中的每一项都成功。 -- 针对新目标端的抽样查询能返回所选每个 `indexScope` 分类的预期历史数据。 -- 受保护的源端 PVC 和 PV 仍然存在且处于绑定状态。 - -如果 phase 为 `Blocked` 或 `Failed`，请勿删除迁移资源、Job 或源端存储卷；请联系支持。 - -如果未创建迁移，请记录「不需要历史数据」的明确决定，并在该决定获批前保留受保护的源端存储卷。 - -### 步骤 6：保留的存储卷 - -受保护的旧 Elasticsearch PVC 和 PV 会被保留。请在迁移和目标端校验完成前保留它们。请勿移除保护注解、删除 PVC/PV 或移除 finalizer。校验完成后如需释放这些存储卷，请联系 Alauda 支持或按照单独的、已批准的清理流程执行。 - -本次升级只保留 ES 的 PVC 和 PV。旧 Kafka 和 ZooKeeper 的存储卷不在保留范围内。 - -请导出或记录最终的 PLF conditions、迁移状态、目标端查询结果以及保留的 PVC/PV 名称，用于实施交接。请勿在本次升级中移除保护注解或删除这些存储卷。 - -## 完成检查清单 - -| 状态 | 期望 | -| --- | --- | -| `PlatformLogForward/platform-default` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | -| `LegacyESMigration/platform-es-history`（如已创建） | `Phase=Succeeded` | -| 新日志查询 | 新的日志、事件和审计记录可以通过新查询链路返回 | -| 旧插件卸载 | `ModuleInfo`、`ClusterPluginInstance/logcenter`、`AppRelease/logcenter` 均不存在，旧 ES 工作负载已消失 | -| 源端存储卷 | 受保护的旧 ES PVC 和 PV 仍然存在 | - -如果任何状态不符合预期，请停止并联系 Alauda 支持。请勿通过删除迁移资源、源端存储卷或目标端数据来绕过故障。 - -## 步骤受阻时 - -- 平台以 `moduleinfo is depended by ...` 拒绝卸载插件：请停止，不要修改 logagent，重新执行受控流程；如果仍然失败，请联系支持。 -- 迁移为 `Blocked`、`Failed` 或一直停留在 `Running`：请勿删除迁移资源、Job、源端存储卷或目标端数据；请联系支持。 -- `ModuleInfo` 或 `ClusterPluginInstance` 重新出现：请先删除 `ClusterPluginInstance`，再删除新的 `ModuleInfo`，并重复 60 秒稳定性检查。 -- `AppRelease/logcenter` 未消失：请勿移除 finalizer；请联系支持。 -- 卸载后新日志查询失败：请停止并联系支持。请勿删除保留的存储卷。 From 7cc13a919441182d95d4488fae7e5d70722910df Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 09:56:30 +0800 Subject: [PATCH 14/38] docs: drop INFO callouts from the ES upgrade guide --- docs/en/upgrade/index.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 69a22ec..a39d355 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -6,10 +6,6 @@ weight: 15 This section explains how to upgrade the Logging components of an existing ACP deployment. -:::info -If the cluster already stores logs in ClickHouse or OpenSearch, no upgrade guide is needed. Follow [Installation](../install_log.mdx) to install or update the plugins. -::: - ## Introduction This guide upgrades a cluster that stores logs with **Alauda Container Platform Log Storage for Elasticsearch** to ACP 4.4, where new log data is written to ClickHouse or to an OpenSearch 3.7.0 cluster. @@ -382,9 +378,7 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ Data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. -:::info The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. -::: Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. From 5c0e06283076e00ae70f99977e02a5ce9ed93a57 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 09:58:16 +0800 Subject: [PATCH 15/38] docs: trim redundant scope and audience guidance --- docs/en/upgrade/index.mdx | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index a39d355..eb1817e 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -34,17 +34,6 @@ Use this guide when the cluster matches the source and target below. | Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | | Supported source Logging plugin versions | 4.2.x, 4.3.x | -Do not use this guide for a fresh installation, or when the cluster already stores logs in ClickHouse or OpenSearch. See [Installation](../install_log.mdx) instead. - -## Who Runs This Procedure - -This is an implementation runbook for Alauda implementation engineers or platform administrators. It is not an end-user self-service procedure. - -- Use a platform account that can access both the global management cluster and the target workload cluster. -- Use a separate terminal or kubectl context for each cluster where possible. Each command block in this guide states which cluster it runs on. -- Before each block, confirm the current context with `kubectl config current-context`. -- In a managed environment, confirm the change window and escalation path with Alauda support before you start. - ## Prerequisites Before you start, ensure that: From b2c647b80e2ca41f5809a44aac9df798a1aeac67 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:06:25 +0800 Subject: [PATCH 16/38] docs: use the product name instead of the internal plugin name --- docs/en/upgrade/index.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index eb1817e..c8fad7f 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -58,11 +58,11 @@ During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafk The operator does not create the external OpenSearch/ClickHouse cluster or the new Kafka service. Create them separately, then provide the connection details through the Secrets in Step 1. -Plan the size with [Log Component Capacity Planning](../architecture/capacity_planning.mdx), and do not plan below the current logcenter deployment. The target must hold the migrated history plus new traffic. The reference disk configuration in that guide is `6000 IOPS` and `250 MB/s` read/write on dedicated SSD mounts; if the actual storage is weaker, move to a larger profile. +Plan the size with [Log Component Capacity Planning](../architecture/capacity_planning.mdx), and do not plan below the current Alauda Container Platform Log Storage for Elasticsearch deployment. The target must hold the migrated history plus new traffic. The reference disk configuration in that guide is `6000 IOPS` and `250 MB/s` read/write on dedicated SSD mounts; if the actual storage is weaker, move to a larger profile. -The legacy logcenter values below are the current deployment baseline, not target settings. They are `chart-alauda-log-center` defaults and can differ per site; check the deployed values before you plan: +The legacy Alauda Container Platform Log Storage for Elasticsearch values below are the current deployment baseline, not target settings. They are `chart-alauda-log-center` defaults and can differ per site; check the deployed values before you plan: -| Legacy logcenter chart setting | Typical baseline | What to prepare for the target | +| Legacy Alauda Container Platform Log Storage for Elasticsearch chart setting | Typical baseline | What to prepare for the target | | --- | --- | --- | | `elasticsearch.storage.node_size`, `node_replicas` | 200 Gi per data node; `node_replicas` sets the data-node count (1 by default; the small-scale profile uses 3) | Size the target storage for the historical data plus new traffic and the target HA policy | | `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | Use independent storage for the new target; the legacy local path is not reused | @@ -86,7 +86,7 @@ Read the target settings from the CRs that are actually deployed in the environm | Check | Expected | | --- | --- | -| Source | Legacy ES, Kafka, ZooKeeper, lanaya, and Razor are running; `logcenter` is `Running` | +| Source | Legacy ES, Kafka, ZooKeeper, lanaya, and Razor are running; Alauda Container Platform Log Storage for Elasticsearch is `Running` | | Target storage | OpenSearch 3.7.0 or the supported ClickHouse topology is reachable; database/Keeper requirements are met | | Target Kafka | New bootstrap is reachable; topics and Logging user access are ready | | Connection Secrets | Required keys are present and valid; legacy Secrets are not reused | @@ -383,7 +383,7 @@ Run this step only when all of these conditions are true: ::: :::warning -This sequence temporarily clears platform discovery fields for `logcenter` so the legacy plugin can be removed. Run it only in the approved change window, and do not modify or delete logagent or its dependencies. +This sequence temporarily clears platform discovery fields for Alauda Container Platform Log Storage for Elasticsearch so the legacy plugin can be removed. Run it only in the approved change window, and do not modify or delete logagent or its dependencies. ::: If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected, or the platform team asks you to run the controlled procedure, use the sequence below. From 8b1e71265dd338e26639cdbe74a9f9ac6b92c8a7 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:08:32 +0800 Subject: [PATCH 17/38] docs: remove internal chart references from the ES upgrade guide --- docs/en/upgrade/index.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index c8fad7f..95ac02e 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -60,17 +60,17 @@ The operator does not create the external OpenSearch/ClickHouse cluster or the n Plan the size with [Log Component Capacity Planning](../architecture/capacity_planning.mdx), and do not plan below the current Alauda Container Platform Log Storage for Elasticsearch deployment. The target must hold the migrated history plus new traffic. The reference disk configuration in that guide is `6000 IOPS` and `250 MB/s` read/write on dedicated SSD mounts; if the actual storage is weaker, move to a larger profile. -The legacy Alauda Container Platform Log Storage for Elasticsearch values below are the current deployment baseline, not target settings. They are `chart-alauda-log-center` defaults and can differ per site; check the deployed values before you plan: +The values in the table are the current deployment baseline, not the target configuration. Actual site values may differ, so verify them before planning: -| Legacy Alauda Container Platform Log Storage for Elasticsearch chart setting | Typical baseline | What to prepare for the target | +| Legacy deployment setting | Typical baseline | What to prepare for the target | | --- | --- | --- | | `elasticsearch.storage.node_size`, `node_replicas` | 200 Gi per data node; `node_replicas` sets the data-node count (1 by default; the small-scale profile uses 3) | Size the target storage for the historical data plus new traffic and the target HA policy | | `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | Use independent storage for the new target; the legacy local path is not reused | | `logging.esReplicas`, `logging.shards` | Index-level settings: 1 replica, per-type shard counts | OpenSearch only: these are the baseline for the index templates; override the template if the target needs different values. They do not map to ClickHouse — ClickHouse uses `externalStorage.shards` and `replicas` for the cluster topology (see the alignment table below) | -| `kafka.retention_hours` | 48 hours | Set the same retention in the target `KafkaTopic` CRs; size the Kafka brokers through your Kafka deployment, not from this legacy chart value | +| `kafka.retention_hours` | 48 hours | Set the same retention in the target `KafkaTopic` CRs; size the Kafka brokers through your Kafka deployment, not from this legacy value | | `logging.ttl` | Logs 7 days; events/audits 180 days; metering 540 days | `PlatformLogForward` applies the same TTLs to the target; plan capacity with the retention you configure there | -The operator applies the TTL and, for ClickHouse, the shard and replica values from the `PlatformLogForward` spec, so the prepared cluster must be able to satisfy them. The chart does not cover the target node size: size ClickHouse with the capacity-planning profiles, and size OpenSearch with your OpenSearch deployment sizing using the same data volume and throughput inputs. +The operator applies the TTL and, for ClickHouse, the shard and replica values from the `PlatformLogForward` spec, so the prepared cluster must be able to satisfy them. These settings do not cover the target node size: size ClickHouse with the capacity-planning profiles, and size OpenSearch with your OpenSearch deployment sizing using the same data volume and throughput inputs. ### Aligning the target with the PlatformLogForward From 82dd97fd8714d71f466fd6b73bb0755b4248fddb Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:15:33 +0800 Subject: [PATCH 18/38] docs: clarify migration creation and data copy order --- docs/en/upgrade/index.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 95ac02e..230a671 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -275,7 +275,7 @@ If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeep Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. -If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin, so the platform can capture the final source state and resolve the source volumes automatically. Creating the migration after the uninstall requires explicit source volume references and cannot use the final capture. +If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. Resource creation and the data copy happen at different times: the platform captures the final source state and records the source volumes when you create the resource, and copies the data only after the uninstall. Creating the migration after the uninstall requires explicit source volume references and cannot use the final capture. #### Target OpenSearch @@ -350,7 +350,7 @@ The standard `indexScope` categories are: If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections cannot be changed after creation. The migration options above show the defaults; adjust them only when your migration plan requires it. -Do not set `source.pvcRefs` in this recommended flow: the platform discovers the source volumes automatically while the legacy Elasticsearch StatefulSet still exists and records them in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. +Do not set `source.pvcRefs` in this flow; the platform records the discovered source volumes in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. Apply the resource and wait for `PrecaptureReady`: From 319e04bf9a42b1c438de7a6511b36a6e97c5b9ce Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:19:56 +0800 Subject: [PATCH 19/38] docs: remove misleading wording in the ES upgrade guide --- docs/en/upgrade/index.mdx | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 230a671..c4ce999 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -41,7 +41,7 @@ Before you start, ensure that: 1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. 2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches use the OpenSearch standard analyzer: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. + - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches fall back to a standard analyzer without Chinese word segmentation: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - The target storage and Kafka connection details prepared according to their product documentation. @@ -66,7 +66,7 @@ The values in the table are the current deployment baseline, not the target conf | --- | --- | --- | | `elasticsearch.storage.node_size`, `node_replicas` | 200 Gi per data node; `node_replicas` sets the data-node count (1 by default; the small-scale profile uses 3) | Size the target storage for the historical data plus new traffic and the target HA policy | | `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | Use independent storage for the new target; the legacy local path is not reused | -| `logging.esReplicas`, `logging.shards` | Index-level settings: 1 replica, per-type shard counts | OpenSearch only: these are the baseline for the index templates; override the template if the target needs different values. They do not map to ClickHouse — ClickHouse uses `externalStorage.shards` and `replicas` for the cluster topology (see the alignment table below) | +| `logging.esReplicas`, `logging.shards` | Index-level settings: 1 replica, per-type shard counts | OpenSearch only: the platform's index templates start at 1 shard and 1 replica and do not inherit these legacy values; override the template if the target needs different values. They do not map to ClickHouse — ClickHouse uses `externalStorage.shards` and `replicas` for the cluster topology (see the alignment table below) | | `kafka.retention_hours` | 48 hours | Set the same retention in the target `KafkaTopic` CRs; size the Kafka brokers through your Kafka deployment, not from this legacy value | | `logging.ttl` | Logs 7 days; events/audits 180 days; metering 540 days | `PlatformLogForward` applies the same TTLs to the target; plan capacity with the retention you configure there | @@ -80,7 +80,7 @@ Read the target settings from the CRs that are actually deployed in the environm | --- | --- | --- | | ClickHouse | `ClickHouseInstallation` CR: `spec.configuration.clusters[].layout.shardsCount` and `replicasCount` | Set `spec.externalStorage.shards` and `replicas` in the `PlatformLogForward` to the values declared in the CHI CR, and set the Secret `cluster` value to the cluster name used by `ON CLUSTER`. The data path expands by these two values, so a mismatch writes to the wrong replica set. | | Kafka | `Kafka` CR (`spec.kafka.config`), `KafkaNodePool` (or `Kafka.spec.kafka` in older layouts) for broker count and storage, `KafkaTopic` CRs, and `KafkaUser` CR | Keep the Secret topic names (`topics.log`, `topics.event`, `topics.audit`) equal to `KafkaTopic.spec.topicName` and to the topic names in the `KafkaUser` ACLs, and keep `kafkaClusterName` equal to the Kafka CR name. Set partitions, replication factor, and retention in the `KafkaTopic` CRs (or the equivalent topic configuration on your Kafka deployment); the replication factor cannot exceed the number of brokers. The broker and topic maximum message size must accept the audit batches — the Kafka default is 1 MiB, which is too small; the reference CRs use 10 MiB. Keep `auto.create.topics.enable` disabled so a misnamed topic is not created automatically. | -| OpenSearch | The deployed OpenSearch cluster (its operator CR or the manifests that run it) and the applied index templates (`GET /_index_template`) | Index shards and replicas are not controlled by the `PlatformLogForward`. The platform applies low-priority templates with 1 shard and 1 replica; if the production node count or HA policy needs different values, apply a higher-priority composable template before cutover and confirm the result with `GET /_index_template`. | +| OpenSearch | The deployed OpenSearch cluster (its operator CR or the manifests that run it) and the applied index templates (`GET /_index_template`) | Index shards and replicas are not controlled by the `PlatformLogForward`. The platform applies low-priority templates with 1 shard and 1 replica; if the production node count or HA policy needs different values, apply a higher-priority composable template before cutover and confirm the result with `GET /_index_template`. Existing indices keep the settings they were created with. | ## Preflight Checklist @@ -127,7 +127,7 @@ stringData: endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first username: "" # Can be omitted when the target allows anonymous access password: "" # Can be omitted when the target allows anonymous access - tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the worker can verify the target before writing + tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the migration can verify the target before writing -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- @@ -254,7 +254,7 @@ The new data path is not available immediately. The platform first creates it, t kubectl get platformlogforward platform-default -w ``` -The `Phase` column changes from `Provisioning` to `Ready`, and the `Ready` column becomes `True` at the same time. Continue only after you see `Ready`. +The `Phase` column reaches `Ready`, and the `Ready` column becomes `True`. Continue only after you see `Ready`. To follow the progress or troubleshoot, read the status conditions: @@ -301,9 +301,9 @@ spec: name: platform-default-os-conn # The same connection Secret as PlatformLogForward namespace: cpaas-system options: - batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 - syncIntervalSeconds: 5 # Minimum wait between batches, in seconds - maxConcurrentJobs: 1 # Concurrent jobs, up to 2 for OpenSearch + batchSize: 250 # Documents written per batch, 1~100000; default 250 + syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (no fixed wait for OpenSearch) + maxConcurrentJobs: 1 # Concurrent jobs; default 1, up to 2 for OpenSearch ``` #### Target ClickHouse @@ -332,9 +332,9 @@ spec: name: platform-default-ch-conn # The same connection Secret as PlatformLogForward namespace: cpaas-system options: - batchSize: 250 # Documents written per batch, 1~100000, defaults to 250 - syncIntervalSeconds: 5 # Minimum wait between batches, in seconds - maxConcurrentJobs: 1 # Concurrent jobs, 1 for ClickHouse only + batchSize: 250 # Documents written per batch, 1~100000; default 250 + syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (1 second for ClickHouse) + maxConcurrentJobs: 1 # Concurrent jobs; ClickHouse is always serial (1) ``` The standard `indexScope` categories are: @@ -348,7 +348,7 @@ The standard `indexScope` categories are: | `event-*` | Kubernetes events | `event-20260825` | | `audit-*` | Audit logs | `audit-20260825` | -If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections cannot be changed after creation. The migration options above show the defaults; adjust them only when your migration plan requires it. +If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections cannot be changed after creation. The `options` block is optional; adjust it only when your migration plan requires it (the example keeps the default batch size and concurrency). Do not set `source.pvcRefs` in this flow; the platform records the discovered source volumes in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. @@ -365,7 +365,7 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` -Data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. +In this flow, data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. @@ -378,7 +378,7 @@ Run this step only when all of these conditions are true: - `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. - If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. -- The platform reports that protection for the legacy Elasticsearch volumes (PVC/PV) is complete. Do not modify or remove this protection; if the platform reports a protection failure, stop and contact support. +- Confirm that the upgrade program has completed protection for the legacy Elasticsearch volumes (PVC/PV). Do not modify or remove this protection; if the platform reports a protection failure, stop and contact support. - If the source uses legacy Kafka, the platform has confirmed that all queued data is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. ::: From 9a9b06c5f250b4a3a0afc7c42140b81abc777aaa Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:26:36 +0800 Subject: [PATCH 20/38] docs: use the ACP 4.4 Log Storage Manager product name --- docs/en/upgrade/index.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index c4ce999..937ea27 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -4,7 +4,7 @@ weight: 15 # Upgrade -This section explains how to upgrade the Logging components of an existing ACP deployment. +This section explains how to upgrade **Alauda Container Platform Log Storage Manager** in an existing ACP deployment. ## Introduction @@ -32,18 +32,18 @@ Use this guide when the cluster matches the source and target below. | Source | A workload cluster with **Alauda Container Platform Log Storage for Elasticsearch** installed and running | | Target platform | ACP 4.4 | | Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | -| Supported source Logging plugin versions | 4.2.x, 4.3.x | +| Supported source plugin versions | 4.2.x, 4.3.x | ## Prerequisites Before you start, ensure that: -1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. -2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. +1. The ACP 4.4 platform upgrade is complete and **Alauda Container Platform Log Storage Manager** can be upgraded. Upgrade its control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. +2. You have downloaded the **Alauda Container Platform Log Storage Manager** package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches fall back to a standard analyzer without Chinese word segmentation: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. - - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. + - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Kafka user granted access to these topics and the related consumer groups. - The target storage and Kafka connection details prepared according to their product documentation. 4. If you migrate historical data, you have the migration image provided for this release, with the complete registry, tag, or digest. Do not reuse an image from an earlier version. 5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. @@ -88,7 +88,7 @@ Read the target settings from the CRs that are actually deployed in the environm | --- | --- | | Source | Legacy ES, Kafka, ZooKeeper, lanaya, and Razor are running; Alauda Container Platform Log Storage for Elasticsearch is `Running` | | Target storage | OpenSearch 3.7.0 or the supported ClickHouse topology is reachable; database/Keeper requirements are met | -| Target Kafka | New bootstrap is reachable; topics and Logging user access are ready | +| Target Kafka | New bootstrap is reachable; topics and Kafka permissions are ready | | Connection Secrets | Required keys are present and valid; legacy Secrets are not reused | | Access and change window | Platform administrator can reach global and workload clusters; support path and change window are confirmed | | Historical migration | The migration image for this release is available; the target account can create and write the required schema and data | @@ -369,7 +369,7 @@ In this flow, data copy starts only after the legacy plugin is uninstalled, so t The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. -Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. +Use the migration image provided for this **Alauda Container Platform Log Storage Manager** release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. ### Step 4: Uninstall the legacy Elasticsearch storage plugin From 59547bf7518df017cce3947afc6454f42b043d87 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:27:43 +0800 Subject: [PATCH 21/38] docs: limit the 4.4 product name to product references --- docs/en/upgrade/index.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 937ea27..848ff65 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -32,7 +32,7 @@ Use this guide when the cluster matches the source and target below. | Source | A workload cluster with **Alauda Container Platform Log Storage for Elasticsearch** installed and running | | Target platform | ACP 4.4 | | Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | -| Supported source plugin versions | 4.2.x, 4.3.x | +| Supported source Logging plugin versions | 4.2.x, 4.3.x | ## Prerequisites @@ -43,7 +43,7 @@ Before you start, ensure that: 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches fall back to a standard analyzer without Chinese word segmentation: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. - - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Kafka user granted access to these topics and the related consumer groups. + - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - The target storage and Kafka connection details prepared according to their product documentation. 4. If you migrate historical data, you have the migration image provided for this release, with the complete registry, tag, or digest. Do not reuse an image from an earlier version. 5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. @@ -88,7 +88,7 @@ Read the target settings from the CRs that are actually deployed in the environm | --- | --- | | Source | Legacy ES, Kafka, ZooKeeper, lanaya, and Razor are running; Alauda Container Platform Log Storage for Elasticsearch is `Running` | | Target storage | OpenSearch 3.7.0 or the supported ClickHouse topology is reachable; database/Keeper requirements are met | -| Target Kafka | New bootstrap is reachable; topics and Kafka permissions are ready | +| Target Kafka | New bootstrap is reachable; topics and Logging user access are ready | | Connection Secrets | Required keys are present and valid; legacy Secrets are not reused | | Access and change window | Platform administrator can reach global and workload clusters; support path and change window are confirmed | | Historical migration | The migration image for this release is available; the target account can create and write the required schema and data | From 08cf0e6dc3c6b7cf27f57d1447fb67c8c53f10a3 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:29:05 +0800 Subject: [PATCH 22/38] docs: keep the source product name in the upgrade introduction --- docs/en/upgrade/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 848ff65..93d0f5f 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -4,7 +4,7 @@ weight: 15 # Upgrade -This section explains how to upgrade **Alauda Container Platform Log Storage Manager** in an existing ACP deployment. +This section explains how to upgrade **Alauda Container Platform Log Storage for Elasticsearch** in an existing ACP deployment. ## Introduction From 37ba92593eb2608c7bf779cee0aba45b8ab4d838 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 10:30:06 +0800 Subject: [PATCH 23/38] docs: roll back the Log Storage Manager renaming --- docs/en/upgrade/index.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 93d0f5f..795d11c 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -38,8 +38,8 @@ Use this guide when the cluster matches the source and target below. Before you start, ensure that: -1. The ACP 4.4 platform upgrade is complete and **Alauda Container Platform Log Storage Manager** can be upgraded. Upgrade its control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. -2. You have downloaded the **Alauda Container Platform Log Storage Manager** package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. +1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. +2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. 3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches fall back to a standard analyzer without Chinese word segmentation: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. @@ -369,7 +369,7 @@ In this flow, data copy starts only after the legacy plugin is uninstalled, so t The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. -Use the migration image provided for this **Alauda Container Platform Log Storage Manager** release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. +Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. ### Step 4: Uninstall the legacy Elasticsearch storage plugin From 1fe9bbeb54325be5e94655742260e42c7152f1fc Mon Sep 17 00:00:00 2001 From: root Date: Wed, 16 Sep 2026 18:53:01 +0800 Subject: [PATCH 24/38] docs: add the environment preparation guide for external log storage Add docs/en/prepare/index.mdx, covering the ClickHouse or OpenSearch and Kafka backends the logging components connect to when they do not create the storage themselves. The guide was validated on an ACP 4.3 cluster, and the following points come from that run: - Every operator must watch cpaas-system, where the storage and messaging resources are created. A namespace-scoped OperatorGroup makes them ignore those resources silently, without status, events, or pods. - The ClickHouseInstallation must not set fsGroup/runAsUser 101, and the logging account name needs backticks because it contains a hyphen. - The Keeper is mandatory for every profile. A three-node quorum uses fully qualified pod names for its raft peers; short pod names do not resolve and the Keeper then never opens its client port. - The RdsKafka controller needs roles: [controller], and its settings are static broker configuration, so they are read from the broker config rather than from kafka-configs.sh. - The RdsKafkaUser readiness condition is status.phase: Active. - The OpenSearch role needs the template permissions at cluster level, plus a role mapping, and analysis-ik is installed through the spec pluginsList fields so that it survives a pod restart. --- docs/en/prepare/index.mdx | 945 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 945 insertions(+) create mode 100644 docs/en/prepare/index.mdx diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx new file mode 100644 index 0000000..b05445e --- /dev/null +++ b/docs/en/prepare/index.mdx @@ -0,0 +1,945 @@ +--- +weight: 13 +--- + +# Environment Preparation + +This chapter explains how to set up the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service that the logging components use: nodes and disks, the operators, the cluster, its accounts, and the Kafka password, user, ACLs, and topics. + +Follow the steps in order and complete each verification. + +## Before You Start + +Make sure you have: + +1. Administrator access to the cluster that runs the logging components. +2. The nodes and disks planned in Step 1. +3. The operator packages available in the platform marketplace: `clickhouse-operator`, the Alauda Kafka operator, and `opensearch-operator`. + +Use [Log Component Capacity Planning](../architecture/capacity_planning.mdx) with the tables below to choose the scale, and place the workloads on dedicated nodes as described in [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). + +## Step 0: Choose the Target and the Scale + +Choose the target (ClickHouse or OpenSearch) and the scale. + +### ClickHouse profiles + +The CPU and memory values are the container limits for each ClickHouse pod. + +| Profile | ClickHouse pods | Layout | CPU limit per pod | Memory limit per pod | Measured throughput | +| --- | --- | --- | --- | --- | --- | +| Single node | 1 | 1 shard × 1 replica | 2C | 4G | 18,000 logs/s | +| Three nodes | 3 | 1 shard × 3 replicas | 2C | 4G | 20,000 logs/s | +| Six nodes | 6 | 2 shards × 3 replicas | 4C | 8G | 40,000 logs/s | +| Nine nodes | 9 | 3 shards × 3 replicas | 4C | 8G | 69,000 logs/s | + +Use the single-node profile for evaluation only. Start production at the three-node profile, and move to six or nine nodes when a single shard no longer fits. + +### Kafka + +Set up three brokers with a 2C/4G limit each. Size the broker volumes by retention and throughput. + +### OpenSearch profiles + +The CPU and memory values are per-node limits. + +| Profile | Nodes | Layout | CPU limit per node | Memory limit per node | Measured throughput | +| --- | --- | --- | --- | --- | --- | +| Small scale | 3 | 3 nodes, all roles | 2C | 4G | 6,300 logs/s | +| Small scale | 5 | 5 nodes, all roles | 2C | 4G | 9,900 logs/s | +| Large scale | 3 + 5 | 3 master, 5 data | 2C master / 8C data | 4G master / 16G data | 25,000 logs/s | +| Large scale | 3 + 7 | 3 master, 7 data | 2C master / 8C data | 4G master / 16G data | 30,000 logs/s | + +Do not size below the smallest profile, and use the large-scale profiles once a single node pool can no longer serve the data volume. If your measured storage is weaker than 6,000 IOPS and 250 MB/s read/write, size up. + +### Disk + +Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, and size the volumes for your retention: 7 days for most logs, 30 days for Kubernetes logs, 180 days for events and audits, and 540 days for metering. The ClickHouse and Kafka examples below use 200 Gi per pod; size the OpenSearch master and data pools separately. + +## Step 1: Nodes and Disks + +1. Select the nodes that will run the storage cluster. Do not co-locate them with business workloads. +2. Label them as infra nodes and add the matching taint, following [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). +3. Attach a dedicated SSD to every node as a persistent path: + - On the traditional operating system layout, use `/cpaas/data/...`. + - On Alauda OS nodes only `/var/cpaas` is writable, so use `/var/cpaas/data/...`. +4. Make sure the path survives node re-provisioning. +5. Create the directories the storage pods use and set their ownership. The examples use the traditional layout; on Alauda OS nodes replace `/cpaas` with `/var/cpaas`. + + ```bash + # ClickHouse runs as uid 101 + sudo mkdir -p /cpaas/data/clickhouse + sudo chown -R 101:101 /cpaas/data/clickhouse + + # Kafka runs as uid 1001 + sudo mkdir -p /cpaas/data/kafka + sudo chown -R 1001:1001 /cpaas/data/kafka + + # OpenSearch runs as uid 1000 + sudo mkdir -p /cpaas/data/opensearch + sudo chown -R 1000:1000 /cpaas/data/opensearch + ``` + +6. Decide how the volumes are provisioned: + +| Approach | When to use it | What you must do | +| --- | --- | --- | +| Static local volumes | You are pinning each pod to a specific node, which is what the infra-node setup usually does | Create one StorageClass without a provisioner, and pre-create one PV per intended pod, each with `nodeAffinity` and `local.path` pointing at the directory above | +| Dynamic provisioner | Your platform provides a block-storage provisioner | Create the StorageClass and let the claims bind dynamically; confirm the provisioner supports `ReadWriteOnce` block volumes and the throughput above | + +Record the StorageClass name; the claims below reference it. + +## Step 2: Install the Operators + +Install the three operators from the platform marketplace. Every storage and messaging resource below is created in `cpaas-system`, so each operator must be able to reconcile resources in that namespace. + +| Operator | Subscription namespace | Namespaces the operator must watch | +| --- | --- | --- | +| `clickhouse-operator` | `cpaas-system` | `cpaas-system` | +| Alauda Kafka operator (`strimzi-kafka-operator`) | `kafka-system` | All namespaces | +| `opensearch-operator` (OpenSearch target only) | `opensearch-operator` | All namespaces | + +- Install only the operators that are missing. If one is already installed on the cluster, for example by an earlier release, keep it and do not install a second copy: two copies of the same operator write to the same `cpaas-system` resources. Check its watch scope instead and widen it if needed. +- Do not create an OperatorGroup in `cpaas-system`. The platform already owns one there, and a second OperatorGroup makes the platform reject every Subscription in that namespace, including its own. +- For `kafka-system` and `opensearch-operator`, the OperatorGroup must have no `spec.targetNamespaces`. If an OperatorGroup scoped to its own namespace already exists, remove the field and wait for the operator pod to restart. + + ```bash + kubectl -n kafka-system patch operatorgroup kafka-system \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + ``` + +Verify each operator before you continue: + +```bash +# ClickHouse +kubectl get crd clickhouseinstallations.clickhouse.altinity.com +kubectl -n cpaas-system get deploy clickhouse-operator + +# Kafka +kubectl get crd rdskafkas.middleware.alauda.io +kubectl -n kafka-system get deploy strimzi-cluster-operator + +# OpenSearch (only when the target is OpenSearch) +kubectl get crd opensearchclusters.opensearch.opster.io +kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager +``` + +:::warning +An operator that does not watch `cpaas-system` ignores the resources below silently: no status, no events, and no pods. Confirm the deployments are ready and that the Kafka and OpenSearch OperatorGroups reach all namespaces before you continue. +::: + +## Step 3: Create the ClickHouse Cluster + +Skip this step when the target is OpenSearch. + +### 3.1 Create the admin password Secret + +```bash +kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 3.2 Create the Keeper (three nodes and above) + +Every profile needs a Keeper, including the single-node profile, because the logging components create `ReplicatedMergeTree` tables. The single-node profile runs the Keeper inside the ClickHouse pod, through the `keeper_server/*` settings in 3.3, so skip this step for it. + +For three nodes and above, create a three-node Keeper quorum and point the `ClickHouseInstallation` at it. The `clickhouse-operator` package on the platform ships only the `ClickHouseInstallation` resources and no Keeper resource, so the Keeper is deployed as its own workload. The `clickhouse-keeper` binary ships in the ClickHouse server image, so use the same `` as the ClickHouse pods. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: cpaas-clickhouse-keeper-config + namespace: cpaas-system +data: + keeper_config.xml: | + + + information + true + + 0.0.0.0 + + 9181 + 1 + /var/lib/clickhouse-keeper/coordination/log + /var/lib/clickhouse-keeper/coordination/snapshots + + 10000 + 30000 + + + 1cpaas-clickhouse-keeper-0.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local9234 + 2cpaas-clickhouse-keeper-1.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local9234 + 3cpaas-clickhouse-keeper-2.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local9234 + + + +--- +apiVersion: v1 +kind: Service +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + clusterIP: None + selector: + app: cpaas-clickhouse-keeper + ports: + - {name: client, port: 9181, targetPort: 9181} + - {name: raft, port: 9234, targetPort: 9234} +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + serviceName: cpaas-clickhouse-keeper + replicas: 3 + selector: + matchLabels: {app: cpaas-clickhouse-keeper} + template: + metadata: + labels: {app: cpaas-clickhouse-keeper} + spec: + containers: + - name: keeper + image: + command: ["/bin/sh", "-c"] + args: + - | + cp /config/keeper_config.xml /tmp/keeper_config.xml + ID=$(( ${HOSTNAME##*-} + 1 )) + sed -i "s|1|${ID}|" /tmp/keeper_config.xml + exec clickhouse-keeper --config-file=/tmp/keeper_config.xml + ports: + - {name: client, containerPort: 9181} + - {name: raft, containerPort: 9234} + resources: + requests: {cpu: 100m, memory: 256Mi} + limits: {cpu: "1", memory: 1Gi} + volumeMounts: + - {name: config, mountPath: /config} + - {name: data, mountPath: /var/lib/clickhouse-keeper} + volumes: + - name: config + configMap: {name: cpaas-clickhouse-keeper-config} + volumeClaimTemplates: + - metadata: {name: data} + spec: + accessModes: [ReadWriteOnce] + storageClassName: # From Step 1 + resources: + requests: {storage: 10Gi} +``` + +The raft peers use fully qualified pod names. Short pod names do not resolve, and a Keeper that cannot reach its peers never opens port `9181`; the `ClickHouseInstallation` then fails to start for no visible reason. + +Wait for one leader and two followers before you continue: + +```bash +for pod in 0 1 2; do + echo "keeper-$pod: $(kubectl -n cpaas-system exec cpaas-clickhouse-keeper-$pod -- \ + clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | + awk '$1 == "zk_server_state" {print $2}')" +done +# expect: one leader, two followers +``` + +### 3.3 Create the ClickHouseInstallation + +Set `shardsCount`, `replicasCount`, and the resource limits from the Step 0 profile, and keep the cluster name `replicated` for the logging components. + +Replace `` with the ClickHouse server image published with the platform middleware packages, for example `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`. + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # The admin password comes from the Secret created above. + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + clusters: + - name: replicated # Reused in the connection Secret + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # From the profile: 1, 1, 2, or 3 + replicasCount: 1 # The example runs one pod; use 3 for three nodes and above + + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + # Co-located Keeper for the single-node profile. For three nodes and above, + # remove these lines and point zookeeper.nodes at your own Keeper quorum. + keeper_server/tcp_port: "9181" + keeper_server/server_id: "1" + keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log + keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots + keeper_server/coordination_settings/operation_timeout_ms: "10000" + keeper_server/coordination_settings/session_timeout_ms: "30000" + keeper_server/raft_configuration/server/id: "1" + keeper_server/raft_configuration/server/hostname: localhost + keeper_server/raft_configuration/server/port: "9234" + + zookeeper: + nodes: + - host: localhost + port: 9181 + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + spec: + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: keeper + containerPort: 9181 + - name: raft + containerPort: 9234 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # From Step 1 +``` + +The `keeper_server/*` settings in the example run the Keeper inside the ClickHouse pod. They belong to the single-node profile: for three nodes and above, delete them and list the Keeper quorum you created in 3.2. + +```yaml + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper-0.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local + port: 9181 + - host: cpaas-clickhouse-keeper-1.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local + port: 9181 + - host: cpaas-clickhouse-keeper-2.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local + port: 9181 +``` + +For three nodes and above, spread the replicas across hosts by replacing the `podDistribution` entry in the pod template with: + +```yaml + podDistribution: + - scope: Replica + topologyKey: kubernetes.io/hostname + type: ReplicaAntiAffinity +``` + +The tables are created inside `observability`; no pre-created database is needed. + +### 3.4 Wait for the cluster + +```bash +kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ + -o jsonpath='{.status.status}{"\n"}' # wait for: Completed + +kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse +``` + +All ClickHouse pods must be `Running` and ready, and every claim must be `Bound`. A StorageClass name that does not exist or cannot bind produces no pods and no error, so check the claims rather than the `ClickHouseInstallation` status alone. + +### 3.5 Create the account used by the logging components + +The account must be able to create and alter tables in the target database. Scope it to that database. + +```sql +CREATE USER `platform-logging` ON CLUSTER 'replicated' IDENTIFIED BY ''; +GRANT ALL ON observability.* TO `platform-logging` ON CLUSTER 'replicated'; +GRANT SYSTEM DROP DNS CACHE ON *.* TO `platform-logging`; +``` + +Backtick the account name: the login name contains a hyphen, and ClickHouse rejects it unquoted. Use the cluster name declared in the `ClickHouseInstallation`. `SYSTEM DROP DNS CACHE` is required by the retention cleaner. On Alauda OS or other FIPS-enabled hosts the password must be at least 32 characters; generate it with `openssl rand -hex 16`. + +### 3.6 Verify + +```bash +CH_POD="$(kubectl -n cpaas-system get pod \ + -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password '' \ + --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" + +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password '' \ + --query "DROP TABLE observability.__perm_check" +``` + +Both commands must succeed; a failed create means the account cannot manage the schema. + +Record these values: + +| Value | Where to read it | +| --- | --- | +| Endpoint | The cluster Service that exposes `8123`, listed by `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=` | +| Cluster name | `spec.configuration.clusters[].name` | +| Database | `spec.configuration.settings.default_database` | +| Shards / replicas | `shardsCount` / `replicasCount` | +| User / password | The account created above | + +## Step 4: Create the Kafka Service + +### 4.1 Create the SASL password Secret + +The password must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts. + +```bash +kubectl -n cpaas-system create secret generic platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 4.2 Create the broker cluster + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + mode: KRaft + version: 4.2.0 # Minimum supported by the Alauda Kafka operator + replicas: 3 + resources: + limits: { cpu: "2", memory: 4Gi } # From the profile + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: + deleteClaim: false + controller: + replicas: 3 + roles: ["controller"] # Required: without it the node pool is rejected + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: + deleteClaim: false + kafka: + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.hours: "48" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # Required: it creates the topics in Step 4.5 + userOperator: {} # Required: it creates the SASL user in Step 4.4 +``` + +Do not omit these settings: + +| Setting | Why it is required | +| --- | --- | +| `message.max.bytes: "10485760"` | Audit batches are about 1.1–1.5 MiB each. The Kafka default of 1 MiB rejects every audit batch. | +| `replica.fetch.max.bytes: "10485760"` | Must be at least `message.max.bytes`, otherwise replica synchronization stalls. | +| `auto.create.topics.enable: "false"` | A mistyped topic name must not be created automatically and silently collect data. | +| `entityOperator.topicOperator` / `userOperator` | Without them, the `RdsTopic` and `RdsKafkaUser` resources in the next steps are not applied to the brokers. | + +### 4.3 Wait for the broker cluster + +```bash +kubectl -n cpaas-system get rdsKafka cpaas-kafka \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' +kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka +``` + +Wait until the `Ready` condition is `True` and all broker pods are `Running`. + +Confirm the required settings reached the brokers: + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ + grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes" /tmp/strimzi.properties +``` + +Both values must be `10485760`, otherwise stop. The broker applies these settings as static configuration, so `kafka-configs.sh --describe` does not show them. + +### 4.4 Create the SASL user and its ACLs + +The logging components use a single account with access to the three topics, the consumer groups, and the broker metadata. + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser +metadata: + name: platform-logging + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + # The three topics + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } + # The consumer groups used by the log pipeline + - host: "*" + operation: All + resource: { type: group, name: alauda_log, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_event, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_audit, patternType: literal } + # The consumer group prefix used by LogForward + - host: "*" + operation: All + resource: { type: group, name: "logforward-", patternType: prefix } + # The consumer group prefix used by the log query service + - host: "*" + operation: All + resource: { type: group, name: "razor-", patternType: prefix } + # Broker metadata + - host: "*" + operation: All + resource: { type: cluster, name: kafka-cluster, patternType: literal } +``` + +All nine entries are required. `operation: All` covers the read and describe permissions these entries need. Verify: + +```bash +kubectl -n cpaas-system get rdskafkauser platform-logging \ + -o jsonpath='{.status.phase}{"\n"}' # expect: Active +kubectl -n cpaas-system get secret platform-logging +``` + +The group names use underscores (`alauda_log`), while the topic names use uppercase letters and underscores (`ALAUDA_LOG_TOPIC`). + +### 4.5 Create the three topics + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC # Broker-side name; must match the ACL and the connection Secret + partitions: 30 # Upper bound for consumer parallelism + replicas: 3 + config: + retention.ms: "172800000" # 48 hours + segment.bytes: "1073741824" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "1073741824" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "1073741824" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +``` + +The resource name must be a valid DNS name. `spec.topicName` is the broker-side name and must match the ACL entries above. + +The example uses a 48-hour retention. Time-based retention alone does not cap disk usage: a burst of traffic can fill the broker volume before the window expires. Either size the broker volumes for the peak rate over the retention window, or add `retention.bytes` to each topic. `retention.bytes` applies per partition, so the broker volumes must hold `partitions × retention.bytes`. + +### 4.6 Verify the Kafka service end to end + +Build the client properties inside the broker pod from the user Secret. The broker listeners require SASL, so every command below needs them. + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ + sh -c 'cat > /tmp/logging-client.properties' <-kafka-bootstrap.cpaas-system.svc:9093` for SASL over TLS, or `:9092` for SASL without TLS | +| Cluster name | `metadata.name` of the `RdsKafka` resource | +| User / password | The `RdsKafkaUser` name and its password | +| Topics | `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, `ALAUDA_AUDIT_TOPIC` | + +## Step 5: Create the OpenSearch Cluster + +Skip this step when the target is ClickHouse. + +### 5.1 Create the cluster + +Set the node pools from the Step 0 profile. The example is 3 + 5: three master nodes and five data nodes. For the small-scale profiles, use a single pool with `roles: [cluster_manager, data]` and `replicas: 3` or `5`. + +```yaml +apiVersion: opensearch.opster.io/v1 +kind: OpenSearchCluster +metadata: + name: cpaas-opensearch + namespace: cpaas-system +spec: + general: + serviceName: cpaas-opensearch + httpPort: 9200 + version: 3.7.0 + security: + tls: + http: + generate: true + transport: + generate: true + perNode: true + nodePools: + - component: masters + replicas: 3 + diskSize: 100Gi + roles: + - cluster_manager + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # From Step 1 + resources: + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + requests: + cpu: "1" + memory: 2Gi + - component: data + replicas: 5 + diskSize: 800Gi + roles: + - data + - ingest + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # From Step 1 + resources: + limits: + cpu: "8" # From the profile + memory: 16Gi # From the profile + requests: + cpu: "2" + memory: 8Gi + dashboards: + replicas: 0 +``` + +Wait for the cluster to be healthy: + +```bash +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' # expect: green +``` + +The health reports `unknown` and then `yellow` while the nodes start and the shards initialize. Wait for `green`. + +### 5.2 Optional: install the Chinese analyzer plugin + +`analysis-ik` is optional. Without it the standard analyzer is used and Chinese text is not segmented; with it Chinese text is segmented. + +Configure the plugin in the `OpenSearchCluster` spec. The operator passes every `pluginsList` entry to `opensearch-plugin install` each time a node starts, so the plugin survives pod restarts and node replacement. Installing it by hand inside a running container does not survive a restart, because the plugin is written outside the data volume. + +| Field | Effect | +| --- | --- | +| `spec.general.pluginsList` | Installs the plugin on every OpenSearch node | +| `spec.bootstrap.pluginsList` | Installs the plugin on the bootstrap pod that forms the cluster. Set it whenever you configure the plugin on a cluster you have not created yet, otherwise cluster initialization can fail. | + +To create the cluster with the plugin from the start, add both fields to the `OpenSearchCluster` of 5.1 and apply that version: + +```yaml +spec: + general: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" + bootstrap: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" +``` + +If you already created the cluster without the plugin, add `spec.general.pluginsList` and re-apply; the operator rolls the nodes to install it. + +The plugin version must match `spec.general.version`. A URL that returns 404 stops every node from starting. On a cluster without external network access, host the zip on an internal server and use that URL. + +Verify that every node has the plugin: + +```bash +for pod in $(kubectl -n cpaas-system get pod \ + -l opster.io/opensearch-cluster=cpaas-opensearch \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "$pod: $(kubectl -n cpaas-system exec "$pod" -c opensearch -- \ + bin/opensearch-plugin list | grep -c '^analysis-ik')" +done +``` + +Each node must print `1`. Then confirm the analyzer segments Chinese text: + +```bash +curl -sk -u "admin:" -X POST "https:///_analyze" \ + -H 'Content-Type: application/json' \ + -d '{"analyzer":"ik_smart","text":"自然语言处理技术"}' +``` + +The response must contain several tokens, for example `自然语言`, `处理`, `技术`. + +### 5.3 Create the account used by the logging components + +The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. Replace `` with the password of the security plugin administrator and `` with a value of at least 32 characters. + +```bash +OS="https://" + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/roles/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{ + "cluster_permissions": [ + "cluster:monitor/*", + "cluster:admin/opendistro/ism/policy/*", + "indices:admin/index_template/put", + "indices:admin/index_template/get", + "indices:admin/template/put", + "indices:admin/template/get" + ], + "index_permissions": [{ + "index_patterns": ["log-*", "event-*", "audit-*", "meter-*"], + "allowed_actions": [ + "indices:admin/create", + "indices:admin/mapping/put", + "indices:data/write/*", + "indices:data/read/*" + ] + }] + }' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/internalusers/platform-logging" \ + -H 'Content-Type: application/json' \ + -d '{"password":"","backend_roles":[]}' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/rolesmapping/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{"users":["platform-logging"]}' +``` + +The template and policy permissions are cluster-level. Declaring `indices:admin/index_template/put` and `indices:admin/template/put` under `index_permissions` does not grant them, and the logging components then fail to install their index templates. Without the role mapping the account is authenticated but has no permissions at all. + +Verify as the new account: + +```bash +curl -sk -u "platform-logging:" -X PUT "$OS/_index_template/perm-check" \ + -H 'Content-Type: application/json' \ + -d '{"index_patterns":["log-perm-check-*"],"template":{"settings":{"number_of_shards":1}}}' + +curl -sk -u "platform-logging:" -X POST "$OS/log-perm-check/_doc" \ + -H 'Content-Type: application/json' -d '{"check":1}' + +curl -sk -u "platform-logging:" "$OS/_index_template/perm-check" +``` + +All three must succeed. A `security_exception` with `no permissions for [...]` means the role or the role mapping is incomplete. Clean up with the administrator account: + +```bash +curl -sk -u "admin:" -X DELETE "$OS/_index_template/perm-check" +curl -sk -u "admin:" -X DELETE "$OS/log-perm-check" +``` + +Record these values: + +| Value | Where to read it | +| --- | --- | +| Endpoint | The service address or load balancer, for example `https://:9200` | +| User / password | The account created above | +| CA certificate | The certificate for the endpoint, when it uses a private CA | + +The platform applies index templates with one shard and one replica. If your HA policy needs different values, apply a composable template with a higher priority and confirm the result with `GET /_index_template`. Existing indices keep the settings they were created with. + +## Step 6: Record the Connection Details + +Record every value below, and confirm the endpoint resolves from the cluster that runs the logging components. + +| Value | ClickHouse | OpenSearch | Kafka | +| --- | --- | --- | --- | +| Endpoint | Required | Required | Required | +| Cluster name | Required | — | Required | +| Database | Required | — | — | +| Shards / replicas | Required | — | — | +| User | Required | Required | Required | +| Password | Required | Required | Required | +| Topics | — | — | Required | +| CA certificate | When the endpoint uses a private CA | When the endpoint uses a private CA (used for historical data migration) | When the endpoint uses a private CA | + +## Environment Checklist + +| Check | Expected | +| --- | --- | +| Nodes and disks | Dedicated nodes labelled, SSD mounted, directories created with the right ownership | +| StorageClass | Exists and binds the volumes the storage cluster uses | +| Operators | ClickHouse, Kafka, and (for OpenSearch) OpenSearch operators are ready, their CRDs exist, and they watch `cpaas-system` | +| ClickHouse | `status.status` is `Completed`, all pods ready, the logging account can create and drop a table. For three nodes and above, the Keeper quorum reports one leader and two followers | +| OpenSearch | Cluster health is `green`, the logging account can manage an index template, and `analysis-ik` is listed on every node and segments Chinese text when you enabled it | +| Kafka brokers | Cluster ready, `message.max.bytes` and `replica.fetch.max.bytes` are both `10485760` | +| Kafka user and ACLs | `RdsKafkaUser` is `Active`, all nine ACL entries are present | +| Kafka topics | The three topics exist on the brokers with the intended partitions and retention | +| Kafka connectivity | A record produced and consumed with the logging account | +| Values recorded | Endpoint, cluster, database, topology, credentials, topics, and CA certificates are all captured | + +Fix any failed check before the logging components connect to this storage. From b5e4c28070b921d115865e91df65befc2c78c87a Mon Sep 17 00:00:00 2001 From: root Date: Wed, 16 Sep 2026 21:43:27 +0800 Subject: [PATCH 25/38] docs: make the environment preparation guide runnable in the field Rewrite the storage, Kafka, and OpenSearch steps so that an implementer can follow the chapter without filling in steps themselves: - Every YAML block now names the file to save it as and the kubectl apply command to run; all commands run from a host with kubectl access. - The ClickHouse section ships two complete manifests, single node and three nodes and above, instead of one manifest plus fragments that had to be merged by hand, and the three-node Keeper quorum has a manifest of its own with three explicit commands to confirm the leader and followers. - The logging account step runs the SQL inside the pod, reading the admin password from the Secret and generating the account password, instead of leaving the reader to work out where to run the SQL. - Static local volumes get a StorageClass and PV example plus the number of PVs each profile needs. - The analyzer plugin is verified with a single-line command and can be enabled on an existing cluster with one patch. Fix contradictions and gaps found while reviewing: - Step 1 taints the infra nodes, but no manifest tolerated that taint, so the storage pods could not land on those nodes. Every manifest now selects node-role.kubernetes.io/infra and tolerates it; the Kafka CRD has no nodeSelector field, so it uses nodeAffinity instead. Verified against the StatefulSets and node pools the three operators generate. - The retention period for Kubernetes logs is 30 days, not 7. - The Kafka controllers were missing from the capacity table, the Kafka CA certificate had no source, and the OpenSearch CA now points at the Secret the operator creates. - Drop the claim about who creates the observability database and the instruction to confirm name resolution from another cluster. Validated on ACP 4.3: 39 command blocks pass bash -n, all manifests pass server-side dry run, and the ClickHouse single node and three node paths, Kafka, and OpenSearch were deployed and exercised end to end. --- docs/en/prepare/index.mdx | 316 ++++++++++++++++++++++++++++++++------ 1 file changed, 273 insertions(+), 43 deletions(-) diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index b05445e..3082625 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -18,6 +18,8 @@ Make sure you have: Use [Log Component Capacity Planning](../architecture/capacity_planning.mdx) with the tables below to choose the scale, and place the workloads on dedicated nodes as described in [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). +Run every command in this chapter from a host that has `kubectl` access to the cluster. Each YAML block is a file you save and then apply; the text below each block gives the file name and the `kubectl apply -f` command. + ## Step 0: Choose the Target and the Scale Choose the target (ClickHouse or OpenSearch) and the scale. @@ -37,7 +39,7 @@ Use the single-node profile for evaluation only. Start production at the three-n ### Kafka -Set up three brokers with a 2C/4G limit each. Size the broker volumes by retention and throughput. +Set up three brokers with a 2C/4G limit each, plus the three controllers that the manifest in Step 4 runs at 1C/2G. Size the broker volumes by retention and throughput. ### OpenSearch profiles @@ -54,12 +56,12 @@ Do not size below the smallest profile, and use the large-scale profiles once a ### Disk -Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, and size the volumes for your retention: 7 days for most logs, 30 days for Kubernetes logs, 180 days for events and audits, and 540 days for metering. The ClickHouse and Kafka examples below use 200 Gi per pod; size the OpenSearch master and data pools separately. +Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, and size the volumes for your retention: 7 days for most logs, 30 days for Kubernetes logs, 180 days for events and audits, and 540 days for metering. The examples below use 200 Gi per ClickHouse pod and per Kafka broker, 20 Gi per Kafka controller, and size the OpenSearch master and data pools separately. ## Step 1: Nodes and Disks 1. Select the nodes that will run the storage cluster. Do not co-locate them with business workloads. -2. Label them as infra nodes and add the matching taint, following [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). +2. Label them as infra nodes and add the matching taint, following [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). The manifests below select `node-role.kubernetes.io/infra` and tolerate that taint; if your cluster uses a different key, make the same change in every manifest. 3. Attach a dedicated SSD to every node as a persistent path: - On the traditional operating system layout, use `/cpaas/data/...`. - On Alauda OS nodes only `/var/cpaas` is writable, so use `/var/cpaas/data/...`. @@ -87,7 +89,46 @@ Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, | Static local volumes | You are pinning each pod to a specific node, which is what the infra-node setup usually does | Create one StorageClass without a provisioner, and pre-create one PV per intended pod, each with `nodeAffinity` and `local.path` pointing at the directory above | | Dynamic provisioner | Your platform provides a block-storage provisioner | Create the StorageClass and let the claims bind dynamically; confirm the provisioner supports `ReadWriteOnce` block volumes and the throughput above | -Record the StorageClass name; the claims below reference it. +For static local volumes, create one StorageClass and one PV per pod, then write the StorageClass name into `` in the manifests below. The number of PVs is the sum of the pods you plan: ClickHouse `shardsCount × replicasCount`, Kafka `replicas + controller.replicas`, and OpenSearch the sum of the node pool `replicas`. + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local +provisioner: kubernetes.io/no-provisioner +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-clickhouse-0 +spec: + capacity: + storage: 200Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local + local: + path: /cpaas/data/clickhouse + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +Save the YAML as `local-storage.yaml` and apply it: + +```bash +kubectl apply -f local-storage.yaml +``` + +Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path`, and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example `/cpaas/data/clickhouse-0` and `/cpaas/data/clickhouse-1`, created with the ownership from step 5. ## Step 2: Install the Operators @@ -205,6 +246,12 @@ spec: metadata: labels: {app: cpaas-clickhouse-keeper} spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule containers: - name: keeper image: @@ -236,25 +283,33 @@ spec: requests: {storage: 10Gi} ``` +Save the YAML as `cpaas-clickhouse-keeper.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse-keeper.yaml +``` + The raft peers use fully qualified pod names. Short pod names do not resolve, and a Keeper that cannot reach its peers never opens port `9181`; the `ClickHouseInstallation` then fails to start for no visible reason. -Wait for one leader and two followers before you continue: +Wait until one Keeper is the leader and the other two are followers before you continue: ```bash -for pod in 0 1 2; do - echo "keeper-$pod: $(kubectl -n cpaas-system exec cpaas-clickhouse-keeper-$pod -- \ - clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | - awk '$1 == "zk_server_state" {print $2}')" -done -# expect: one leader, two followers +kubectl -n cpaas-system get pod -l app=cpaas-clickhouse-keeper +kubectl -n cpaas-system exec cpaas-clickhouse-keeper-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec cpaas-clickhouse-keeper-1 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec cpaas-clickhouse-keeper-2 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state ``` +The three lines must report one `leader` and two `follower`. + ### 3.3 Create the ClickHouseInstallation -Set `shardsCount`, `replicasCount`, and the resource limits from the Step 0 profile, and keep the cluster name `replicated` for the logging components. +Keep the cluster name `replicated` for the logging components, and set `shardsCount` and `replicasCount` from the Step 0 profile. Apply one of the two manifests below, depending on the profile you chose. Replace `` with the ClickHouse server image published with the platform middleware packages, for example `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`. +**Single node.** The Keeper runs inside the ClickHouse pod through the `keeper_server/*` settings. + ```yaml apiVersion: clickhouse.altinity.com/v1 kind: ClickHouseInstallation @@ -321,6 +376,12 @@ spec: topologyKey: kubernetes.io/hostname type: ShardAntiAffinity spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule containers: - name: clickhouse image: @@ -367,9 +428,49 @@ spec: storageClassName: # From Step 1 ``` -The `keeper_server/*` settings in the example run the Keeper inside the ClickHouse pod. They belong to the single-node profile: for three nodes and above, delete them and list the Keeper quorum you created in 3.2. +Save the YAML as `cpaas-clickhouse.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +**Three nodes and above.** Use this manifest instead of the one above. It drops the Keeper settings, points `zookeeper.nodes` at the three Keeper pods created in 3.2, and spreads the replicas across hosts. ```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + clusters: + - name: replicated # Reused in the connection Secret + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # From the profile: 1, 2, or 3 + replicasCount: 3 # 3 for three nodes and above + + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + zookeeper: nodes: - host: cpaas-clickhouse-keeper-0.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local @@ -378,24 +479,82 @@ The `keeper_server/*` settings in the example run the Keeper inside the ClickHou port: 9181 - host: cpaas-clickhouse-keeper-2.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local port: 9181 -``` -For three nodes and above, spread the replicas across hosts by replacing the `podDistribution` entry in the pod template with: + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template -```yaml + templates: + podTemplates: + - name: pod-template podDistribution: - scope: Replica topologyKey: kubernetes.io/hostname type: ReplicaAntiAffinity + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # From Step 1 ``` -The tables are created inside `observability`; no pre-created database is needed. +Save the YAML as `cpaas-clickhouse.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +`default_database: observability` makes ClickHouse create the `observability` database at startup, so there is no database to create here. ### 3.4 Wait for the cluster ```bash kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ - -o jsonpath='{.status.status}{"\n"}' # wait for: Completed + -o jsonpath='{.status.status}{"\n"}' # re-run until it reports: Completed kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse @@ -408,27 +567,36 @@ All ClickHouse pods must be `Running` and ready, and every claim must be `Bound` The account must be able to create and alter tables in the target database. Scope it to that database. -```sql -CREATE USER `platform-logging` ON CLUSTER 'replicated' IDENTIFIED BY ''; -GRANT ALL ON observability.* TO `platform-logging` ON CLUSTER 'replicated'; -GRANT SYSTEM DROP DNS CACHE ON *.* TO `platform-logging`; -``` - -Backtick the account name: the login name contains a hyphen, and ClickHouse rejects it unquoted. Use the cluster name declared in the `ClickHouseInstallation`. `SYSTEM DROP DNS CACHE` is required by the retention cleaner. On Alauda OS or other FIPS-enabled hosts the password must be at least 32 characters; generate it with `openssl rand -hex 16`. - -### 3.6 Verify +Run these on the host. They generate the password, then run the SQL inside the ClickHouse pod. The account name needs backticks because it contains a hyphen, and `SYSTEM DROP DNS CACHE` is required by the retention cleaner. Record the printed password: the connection details in Step 6 need it, and 3.6 reuses the `LOG_PASSWORD` variable. ```bash CH_POD="$(kubectl -n cpaas-system get pod \ -l clickhouse.altinity.com/chi=cpaas-clickhouse \ -o jsonpath='{.items[0].metadata.name}')" +ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-basic-auth \ + -o jsonpath='{.data.password}' | base64 -d)" +LOG_PASSWORD="$(openssl rand -hex 16)" +echo "platform-logging password: $LOG_PASSWORD" + +kubectl -n cpaas-system exec -i "$CH_POD" -- clickhouse-client \ + --user admin --password "$ADMIN_PASSWORD" --multiquery <' \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" kubectl -n cpaas-system exec "$CH_POD" -- \ - clickhouse-client --user platform-logging --password '' \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ --query "DROP TABLE observability.__perm_check" ``` @@ -438,7 +606,7 @@ Record these values: | Value | Where to read it | | --- | --- | -| Endpoint | The cluster Service that exposes `8123`, listed by `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=` | +| Endpoint | The cluster Service that exposes `8123`, listed by `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse` | | Cluster name | `spec.configuration.clusters[].name` | | Database | `spec.configuration.settings.default_database` | | Shards / replicas | `shardsCount` / `replicasCount` | @@ -477,6 +645,19 @@ spec: controller: replicas: 3 roles: ["controller"] # Required: without it the node pool is rejected + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule resources: limits: { cpu: "1", memory: 2Gi } requests: { cpu: 100m, memory: 512Mi } @@ -485,6 +666,19 @@ spec: class: deleteClaim: false kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule listeners: plain: authentication: @@ -511,6 +705,12 @@ spec: userOperator: {} # Required: it creates the SASL user in Step 4.4 ``` +Save the YAML as `cpaas-kafka.yaml` and apply it: + +```bash +kubectl apply -f cpaas-kafka.yaml +``` + Do not omit these settings: | Setting | Why it is required | @@ -600,6 +800,12 @@ spec: resource: { type: cluster, name: kafka-cluster, patternType: literal } ``` +Save the YAML as `platform-logging-user.yaml` and apply it: + +```bash +kubectl apply -f platform-logging-user.yaml +``` + All nine entries are required. `operation: All` covers the read and describe permissions these entries need. Verify: ```bash @@ -668,6 +874,12 @@ spec: max.message.bytes: "10485760" ``` +Save the YAML as `alauda-topics.yaml` and apply it: + +```bash +kubectl apply -f alauda-topics.yaml +``` + The resource name must be a valid DNS name. `spec.topicName` is the broker-side name and must match the ACL entries above. The example uses a 48-hour retention. Time-based retention alone does not cap disk usage: a burst of traffic can fill the broker volume before the window expires. Either size the broker volumes for the peak rate over the retention window, or add `retention.bytes` to each topic. `retention.bytes` applies per partition, so the broker volumes must hold `partitions × retention.bytes`. @@ -720,6 +932,7 @@ Record these values: | Cluster name | `metadata.name` of the `RdsKafka` resource | | User / password | The `RdsKafkaUser` name and its password | | Topics | `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, `ALAUDA_AUDIT_TOPIC` | +| CA certificate | Only for the TLS listener on `9093`. The Kafka operator publishes it in `cpaas-system` as a Secret whose name ends with `-cluster-ca-cert` | ## Step 5: Create the OpenSearch Cluster @@ -753,6 +966,12 @@ spec: diskSize: 100Gi roles: - cluster_manager + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule persistence: pvc: accessModes: @@ -771,6 +990,12 @@ spec: roles: - data - ingest + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule persistence: pvc: accessModes: @@ -787,6 +1012,12 @@ spec: replicas: 0 ``` +Save the YAML as `cpaas-opensearch.yaml` and apply it: + +```bash +kubectl apply -f cpaas-opensearch.yaml +``` + Wait for the cluster to be healthy: ```bash @@ -807,7 +1038,7 @@ Configure the plugin in the `OpenSearchCluster` spec. The operator passes every | `spec.general.pluginsList` | Installs the plugin on every OpenSearch node | | `spec.bootstrap.pluginsList` | Installs the plugin on the bootstrap pod that forms the cluster. Set it whenever you configure the plugin on a cluster you have not created yet, otherwise cluster initialization can fail. | -To create the cluster with the plugin from the start, add both fields to the `OpenSearchCluster` of 5.1 and apply that version: +To create the cluster with the plugin from the start, add both fields to `cpaas-opensearch.yaml` from 5.1 before you apply it: ```yaml spec: @@ -819,22 +1050,21 @@ spec: - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" ``` -If you already created the cluster without the plugin, add `spec.general.pluginsList` and re-apply; the operator rolls the nodes to install it. +If you already created the cluster without the plugin, patch it instead. The operator rolls the nodes to install the plugin: + +```bash +kubectl -n cpaas-system patch opensearchcluster cpaas-opensearch --type=merge -p '{"spec":{"general":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]},"bootstrap":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]}}}' +``` The plugin version must match `spec.general.version`. A URL that returns 404 stops every node from starting. On a cluster without external network access, host the zip on an internal server and use that URL. Verify that every node has the plugin: ```bash -for pod in $(kubectl -n cpaas-system get pod \ - -l opster.io/opensearch-cluster=cpaas-opensearch \ - -o jsonpath='{.items[*].metadata.name}'); do - echo "$pod: $(kubectl -n cpaas-system exec "$pod" -c opensearch -- \ - bin/opensearch-plugin list | grep -c '^analysis-ik')" -done +for p in $(kubectl -n cpaas-system get pod -l opster.io/opensearch-cluster=cpaas-opensearch -o jsonpath='{.items[*].metadata.name}'); do echo -n "$p: "; kubectl -n cpaas-system exec $p -c opensearch -- bin/opensearch-plugin list | grep -c '^analysis-ik'; done ``` -Each node must print `1`. Then confirm the analyzer segments Chinese text: +Each node must print `1`. Then confirm the analyzer segments Chinese text. Replace `` with the cluster service address, for example `https://cpaas-opensearch.cpaas-system.svc:9200`: ```bash curl -sk -u "admin:" -X POST "https:///_analyze" \ @@ -849,7 +1079,7 @@ The response must contain several tokens, for example `自然语言`, `处理`, The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. Replace `` with the password of the security plugin administrator and `` with a value of at least 32 characters. ```bash -OS="https://" +OS="https://" # the cluster service, for example https://cpaas-opensearch.cpaas-system.svc:9200 curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/roles/log_storage_writer" \ -H 'Content-Type: application/json' -d '{ @@ -908,13 +1138,13 @@ Record these values: | --- | --- | | Endpoint | The service address or load balancer, for example `https://:9200` | | User / password | The account created above | -| CA certificate | The certificate for the endpoint, when it uses a private CA | +| CA certificate | Only when the endpoint uses a private CA. The operator creates it in `cpaas-system` as Secret `cpaas-opensearch-ca` | The platform applies index templates with one shard and one replica. If your HA policy needs different values, apply a composable template with a higher priority and confirm the result with `GET /_index_template`. Existing indices keep the settings they were created with. ## Step 6: Record the Connection Details -Record every value below, and confirm the endpoint resolves from the cluster that runs the logging components. +Record every value below. | Value | ClickHouse | OpenSearch | Kafka | | --- | --- | --- | --- | @@ -931,7 +1161,7 @@ Record every value below, and confirm the endpoint resolves from the cluster tha | Check | Expected | | --- | --- | -| Nodes and disks | Dedicated nodes labelled, SSD mounted, directories created with the right ownership | +| Nodes and disks | Dedicated nodes labelled and tainted, SSD mounted, directories created with the right ownership, and the manifests tolerate that taint | | StorageClass | Exists and binds the volumes the storage cluster uses | | Operators | ClickHouse, Kafka, and (for OpenSearch) OpenSearch operators are ready, their CRDs exist, and they watch `cpaas-system` | | ClickHouse | `status.status` is `Completed`, all pods ready, the logging account can create and drop a table. For three nodes and above, the Keeper quorum reports one leader and two followers | From c174ff4717933b95f5c0913a5a4dc64754b1450f Mon Sep 17 00:00:00 2001 From: root Date: Wed, 16 Sep 2026 23:10:29 +0800 Subject: [PATCH 26/38] docs: run the ClickHouse Keeper inside the ClickHouseInstallation The ClickHouse operator in the platform ships only ClickHouseInstallation, ClickHouseInstallationTemplate and ClickHouseOperatorConfiguration. There is no Keeper custom resource, and a ClickHouseInstallation cannot declare a Keeper quorum either: its schema has no Keeper field, and a raft_configuration placed under configuration.settings is dropped, leaving every pod with the same server_id. Deploy the Keeper the way the Keeper-in-ClickHouseInstallation procedure does instead, so that everything stays operator-managed and no raw workload is created: - 3.2 now only creates the headless Keeper client Service. - The three-node manifest keeps the static Keeper configuration in the cluster files, pulls in a generated file with include_from, and lets an init container write the per-pod server_id and raft member list. - The readiness probe checks the Raft port. This is required: pod creation is serialized, and with the default HTTP probe the first replica can never form a quorum, so ClickHouse never reports ready and the remaining replicas are never created. - A wait-for-self-dns init container guards the distributed DDL worker against a pod that starts before its headless Service DNS record exists. Also fixed while testing: the three-node manifest was missing the users block, so the admin account did not exist and step 3.5 could not work, and it defined no service template, which made the operator create a LoadBalancer Service whose external address stays pending. Verified on a clean deployment: the installation reaches Completed, the Keeper reports one leader and two followers, system.zookeeper_connection points at the Keeper Service, CREATE TABLE ON CLUSTER reaches all three replicas, a ReplicatedMergeTree row written on one replica is readable on all three, and the account steps run as written. --- docs/en/prepare/index.mdx | 284 ++++++++++++++++++++++---------------- 1 file changed, 162 insertions(+), 122 deletions(-) diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 3082625..08248e5 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -182,125 +182,40 @@ kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ --from-literal=password="$(openssl rand -hex 16)" ``` -### 3.2 Create the Keeper (three nodes and above) +### 3.2 Create the Keeper client Service (three nodes and above) -Every profile needs a Keeper, including the single-node profile, because the logging components create `ReplicatedMergeTree` tables. The single-node profile runs the Keeper inside the ClickHouse pod, through the `keeper_server/*` settings in 3.3, so skip this step for it. +Every profile needs a Keeper, including the single-node profile, because the logging components create `ReplicatedMergeTree` tables. The single-node profile runs the Keeper inside its ClickHouse pod through the `keeper_server/*` settings in 3.3, so skip this step for it. -For three nodes and above, create a three-node Keeper quorum and point the `ClickHouseInstallation` at it. The `clickhouse-operator` package on the platform ships only the `ClickHouseInstallation` resources and no Keeper resource, so the Keeper is deployed as its own workload. The `clickhouse-keeper` binary ships in the ClickHouse server image, so use the same `` as the ClickHouse pods. +For three nodes and above, the ClickHouse pods themselves form the Keeper quorum: every ClickHouse pod is also a Keeper member. ClickHouse reaches that quorum through a headless Service that selects all ready pods of the installation. + +Save the YAML as `cpaas-clickhouse-keeper-service.yaml` and apply it: ```yaml apiVersion: v1 -kind: ConfigMap -metadata: - name: cpaas-clickhouse-keeper-config - namespace: cpaas-system -data: - keeper_config.xml: | - - - information - true - - 0.0.0.0 - - 9181 - 1 - /var/lib/clickhouse-keeper/coordination/log - /var/lib/clickhouse-keeper/coordination/snapshots - - 10000 - 30000 - - - 1cpaas-clickhouse-keeper-0.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local9234 - 2cpaas-clickhouse-keeper-1.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local9234 - 3cpaas-clickhouse-keeper-2.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local9234 - - - ---- -apiVersion: v1 kind: Service metadata: name: cpaas-clickhouse-keeper namespace: cpaas-system spec: clusterIP: None - selector: - app: cpaas-clickhouse-keeper + type: ClusterIP ports: - - {name: client, port: 9181, targetPort: 9181} - - {name: raft, port: 9234, targetPort: 9234} ---- -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: cpaas-clickhouse-keeper - namespace: cpaas-system -spec: - serviceName: cpaas-clickhouse-keeper - replicas: 3 + - name: keeper + port: 9181 + protocol: TCP + targetPort: 9181 selector: - matchLabels: {app: cpaas-clickhouse-keeper} - template: - metadata: - labels: {app: cpaas-clickhouse-keeper} - spec: - nodeSelector: - node-role.kubernetes.io/infra: "" # The label you set in Step 1 - tolerations: - - key: node-role.kubernetes.io/infra # The taint you set in Step 1 - operator: Exists - effect: NoSchedule - containers: - - name: keeper - image: - command: ["/bin/sh", "-c"] - args: - - | - cp /config/keeper_config.xml /tmp/keeper_config.xml - ID=$(( ${HOSTNAME##*-} + 1 )) - sed -i "s|1|${ID}|" /tmp/keeper_config.xml - exec clickhouse-keeper --config-file=/tmp/keeper_config.xml - ports: - - {name: client, containerPort: 9181} - - {name: raft, containerPort: 9234} - resources: - requests: {cpu: 100m, memory: 256Mi} - limits: {cpu: "1", memory: 1Gi} - volumeMounts: - - {name: config, mountPath: /config} - - {name: data, mountPath: /var/lib/clickhouse-keeper} - volumes: - - name: config - configMap: {name: cpaas-clickhouse-keeper-config} - volumeClaimTemplates: - - metadata: {name: data} - spec: - accessModes: [ReadWriteOnce] - storageClassName: # From Step 1 - resources: - requests: {storage: 10Gi} + clickhouse.altinity.com/chi: cpaas-clickhouse + clickhouse.altinity.com/namespace: cpaas-system + clickhouse.altinity.com/ready: "yes" + clickhouse.altinity.com/role: keeper ``` -Save the YAML as `cpaas-clickhouse-keeper.yaml` and apply it: - -```bash -kubectl apply -f cpaas-clickhouse-keeper.yaml -``` - -The raft peers use fully qualified pod names. Short pod names do not resolve, and a Keeper that cannot reach its peers never opens port `9181`; the `ClickHouseInstallation` then fails to start for no visible reason. - -Wait until one Keeper is the leader and the other two are followers before you continue: - ```bash -kubectl -n cpaas-system get pod -l app=cpaas-clickhouse-keeper -kubectl -n cpaas-system exec cpaas-clickhouse-keeper-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -kubectl -n cpaas-system exec cpaas-clickhouse-keeper-1 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -kubectl -n cpaas-system exec cpaas-clickhouse-keeper-2 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl apply -f cpaas-clickhouse-keeper-service.yaml ``` -The three lines must report one `leader` and two `follower`. +The `chi`, `namespace` and `ready` labels are set by the operator. The `role: keeper` label comes from the pod template in 3.3. ### 3.3 Create the ClickHouseInstallation @@ -346,7 +261,7 @@ spec: default_database: observability # Reused in the connection Secret merge_tree/materialize_ttl_recalculate_only: "1" # Co-located Keeper for the single-node profile. For three nodes and above, - # remove these lines and point zookeeper.nodes at your own Keeper quorum. + # use the manifest below instead, which runs Keeper in every ClickHouse pod. keeper_server/tcp_port: "9181" keeper_server/server_id: "1" keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log @@ -434,7 +349,13 @@ Save the YAML as `cpaas-clickhouse.yaml` and apply it: kubectl apply -f cpaas-clickhouse.yaml ``` -**Three nodes and above.** Use this manifest instead of the one above. It drops the Keeper settings, points `zookeeper.nodes` at the three Keeper pods created in 3.2, and spreads the replicas across hosts. +**Three nodes and above.** Use this manifest instead of the one above. The Keeper runs inside every ClickHouse pod, so the pods form the quorum among themselves and the installation stays a single `ClickHouseInstallation`. + +The static Keeper configuration is injected through the cluster `files` and pulls in a generated file with `include_from`. The identity-dependent part — `server_id` and the member list — is generated per pod by an init container into an in-memory `emptyDir`. Keep `SHARDS_COUNT` and `REPLICAS_COUNT` in that init container equal to `layout.shardsCount` and `layout.replicasCount`, otherwise the member list is incomplete and the quorum never forms. + +The readiness probe checks the Raft port. This is required: the default HTTP probe only succeeds after ClickHouse is serving, and ClickHouse does not finish starting until the Keeper quorum exists, so the operator would wait for the first replica forever and never create the remaining ones. + +The `wait-for-self-dns` init container waits until the pod resolves its own headless service name. Without it, a pod that starts before its DNS record is published initialises its distributed DDL worker against an unresolved hostname and then never retries: `CREATE TABLE ... ON CLUSTER` succeeds on the other replicas, and that replica silently misses the statement. ```yaml apiVersion: clickhouse.altinity.com/v1 @@ -445,6 +366,7 @@ metadata: spec: configuration: users: + # The admin password comes from the Secret created above. admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password admin/networks/ip: - "0.0.0.0/0" @@ -458,6 +380,15 @@ spec: default/max_execution_time: 120 default/max_estimated_execution_time: 120 + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper # The Service created in 3.2 + port: 9181 + + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + clusters: - name: replicated # Reused in the connection Secret templates: @@ -466,33 +397,35 @@ spec: layout: shardsCount: 1 # From the profile: 1, 2, or 3 replicasCount: 3 # 3 for three nodes and above - - settings: - default_database: observability # Reused in the connection Secret - merge_tree/materialize_ttl_recalculate_only: "1" - - zookeeper: - nodes: - - host: cpaas-clickhouse-keeper-0.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local - port: 9181 - - host: cpaas-clickhouse-keeper-1.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local - port: 9181 - - host: cpaas-clickhouse-keeper-2.cpaas-clickhouse-keeper.cpaas-system.svc.cluster.local - port: 9181 - + shards: + - files: + keeper_config.xml: | + + /tmp/clickhouse/keeper_dynamic_configuration.xml + + /var/lib/clickhouse-keeper + 9181 + * + + information + + + defaults: templates: podTemplate: pod-template dataVolumeClaimTemplate: data-volumeclaim-template serviceTemplate: service-template - templates: podTemplates: - name: pod-template podDistribution: - - scope: Replica + - scope: Shard topologyKey: kubernetes.io/hostname - type: ReplicaAntiAffinity + type: ShardAntiAffinity + metadata: + labels: + clickhouse.altinity.com/role: keeper # Selected by the Service in 3.2 spec: nodeSelector: node-role.kubernetes.io/infra: "" # The label you set in Step 1 @@ -503,6 +436,9 @@ spec: containers: - name: clickhouse image: + env: + - name: RAFT_PORT + value: "9444" ports: - name: http containerPort: 8123 @@ -510,6 +446,10 @@ spec: containerPort: 9000 - name: interserver containerPort: 9009 + - name: ch-keeper + containerPort: 9181 + - name: raft + containerPort: 9444 resources: requests: cpu: "1" @@ -520,7 +460,92 @@ spec: volumeMounts: - name: data-volumeclaim-template mountPath: /var/lib/clickhouse - + - name: keeper-dynamic-config + mountPath: /tmp/clickhouse + readinessProbe: + tcpSocket: + port: 9444 + initialDelaySeconds: 10 + timeoutSeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + initContainers: + - name: wait-for-self-dns + image: + command: + - /bin/bash + - -c + - | + SELF="$(hostname -s).$(hostname -d)" + MY_IP="$(hostname -i)" + for i in $(seq 1 60); do + RESOLVED=$(getent hosts "$SELF" 2>/dev/null | awk '{print $1}' | head -1) + if [ "$RESOLVED" = "$MY_IP" ]; then + echo "resolved $SELF to $MY_IP after ${i}s" + exit 0 + fi + sleep 2 + done + echo "still unresolved after 120s, continuing" + exit 0 + - name: keeper-config-initializer + image: + env: + - name: RAFT_PORT + value: "9444" + - name: SHARDS_COUNT + value: "1" # Keep equal to layout.shardsCount + - name: REPLICAS_COUNT + value: "3" # Keep equal to layout.replicasCount + command: + - /bin/bash + - -c + - | + set -euo pipefail + OUT="/tmp/config/keeper_dynamic_configuration.xml" + HOST=$(hostname -s) + DOMAIN=$(hostname -d) + if [[ $HOST =~ (.*)-([0-9]+)-([0-9]+)-([0-9]+)$ ]]; then + SHARD=${BASH_REMATCH[2]} + REPLICA=${BASH_REMATCH[3]} + else + echo "Failed to parse shard/replica from hostname $HOST"; exit 1 + fi + if [[ $DOMAIN =~ ^(.*)-([0-9]+)-([0-9]+)\.(.*)$ ]]; then + DOMAIN_NAME=${BASH_REMATCH[1]} + DOMAIN_SUFFIX=.${BASH_REMATCH[4]} + else + echo "Failed to parse domain $DOMAIN"; exit 1 + fi + MY_ID=$((SHARD * REPLICAS_COUNT + REPLICA + 1)) + KEEPER_ID=1 + { + echo "" + echo " " + echo " ${MY_ID}" + echo " " + for (( i=0; i" + echo " ${KEEPER_ID}" + echo " ${DOMAIN_NAME}-${i}-${j}${DOMAIN_SUFFIX}" + echo " ${RAFT_PORT}" + echo " " + KEEPER_ID=$((KEEPER_ID + 1)) + done + done + echo " " + echo " " + echo "" + } > "$OUT" + echo "Keeper dynamic configuration generated for server_id=${MY_ID}" + volumeMounts: + - name: keeper-dynamic-config + mountPath: /tmp/config + volumes: + - name: keeper-dynamic-config + emptyDir: + medium: Memory serviceTemplates: - name: service-template spec: @@ -530,7 +555,6 @@ spec: - name: tcp port: 9000 type: ClusterIP - volumeClaimTemplates: - name: data-volumeclaim-template spec: @@ -563,6 +587,22 @@ kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse All ClickHouse pods must be `Running` and ready, and every claim must be `Bound`. A StorageClass name that does not exist or cannot bind produces no pods and no error, so check the claims rather than the `ClickHouseInstallation` status alone. +For three nodes and above, confirm the Keeper quorum before you continue. The Keeper runs inside the ClickHouse pods, so check one pod per replica: + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +``` + +The three lines must report one `leader` and two `follower`. Also confirm that ClickHouse reads the quorum through the Service from 3.2: + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# expect: cpaas-clickhouse-keeper 9181 +``` + ### 3.5 Create the account used by the logging components The account must be able to create and alter tables in the target database. Scope it to that database. From bec0c2d9dd0725ba7e0cb2492e452a9ef86d77f2 Mon Sep 17 00:00:00 2001 From: root Date: Wed, 16 Sep 2026 23:25:45 +0800 Subject: [PATCH 27/38] docs: keep ClickHouse Keeper state on the persistent volume Reviewing the ClickHouse steps against the Keeper-in-ClickHouseInstallation procedure turned up two gaps. The Keeper path sat at /var/lib/clickhouse-keeper while the data volume is mounted at /var/lib/clickhouse, so the Keeper log and snapshots landed in the container filesystem and were lost on every pod restart. The procedure itself asks for that path to be on the same persistent volume, which is also what the operator's own fixture does, so the path now sits under the mount as /var/lib/clickhouse/coordination. Verified by writing a marker inside the pod and reading it back from the local volume on the node. The manifest also omitted the self-observability system table TTLs that the procedure sets. Those tables grow without bound and eventually fill the data volume, so asynchronous_metric_log, metric_log and trace_log now carry the 7-day delete TTL on both the single-node and the replicated manifests. Redeployed from scratch and re-verified: the installation reaches Completed, Keeper reports one leader and two followers, CREATE TABLE ON CLUSTER reaches all three replicas, a replicated row written on one replica is readable on all three, and the account steps run as written. --- docs/en/prepare/index.mdx | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 08248e5..4591a1f 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -260,6 +260,16 @@ spec: settings: default_database: observability # Reused in the connection Secret merge_tree/materialize_ttl_recalculate_only: "1" + # Self-observability system tables grow without bound and eventually fill the volume. + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" # Co-located Keeper for the single-node profile. For three nodes and above, # use the manifest below instead, which runs Keeper in every ClickHouse pod. keeper_server/tcp_port: "9181" @@ -355,6 +365,8 @@ The static Keeper configuration is injected through the cluster `files` and pull The readiness probe checks the Raft port. This is required: the default HTTP probe only succeeds after ClickHouse is serving, and ClickHouse does not finish starting until the Keeper quorum exists, so the operator would wait for the first replica forever and never create the remaining ones. +The Keeper `path` sits under `/var/lib/clickhouse`, which is the mounted data volume, so the Keeper log and snapshots live on the persistent volume together with the ClickHouse data. Do not move it outside that mount: Keeper state kept in the container filesystem is lost whenever the pod restarts. + The `wait-for-self-dns` init container waits until the pod resolves its own headless service name. Without it, a pod that starts before its DNS record is published initialises its distributed DDL worker against an unresolved hostname and then never retries: `CREATE TABLE ... ON CLUSTER` succeeds on the other replicas, and that replica silently misses the statement. ```yaml @@ -388,6 +400,16 @@ spec: settings: default_database: observability # Reused in the connection Secret merge_tree/materialize_ttl_recalculate_only: "1" + # Self-observability system tables grow without bound and eventually fill the volume. + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" clusters: - name: replicated # Reused in the connection Secret @@ -403,7 +425,7 @@ spec: /tmp/clickhouse/keeper_dynamic_configuration.xml - /var/lib/clickhouse-keeper + /var/lib/clickhouse/coordination 9181 * From ba645b7c6e11efee32bf52697a35c5271e7c694e Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 07:44:31 +0800 Subject: [PATCH 28/38] docs: close the OpenSearch and Kafka gaps against the knowledge base Compared this chapter with the OpenSearch and Kafka articles in the platform knowledge base and fixed what they cover and we did not. OpenSearch: - The account steps authenticate as `admin` but never said where that password comes from. State that the operator stores it in the `-admin-password` Secret, that `admin` / `admin` is the default the examples assume, give the command that reads it, and warn that the password has to be changed together with the operator or the operator's own health checks break. - Note that OpenSearch needs `vm.max_map_count` at 262144, that the operator sets it from an init container, and that only Pod Security Admission restricted namespaces need it set on the nodes by hand. Kafka: - Add `auto.leader.rebalance.enable: "false"`, which the knowledge base lists as a recommended default so partition leaders are not moved without an operator deciding to. - State that the operator applies hard pod anti-affinity and add the command that shows the three brokers on three distinct nodes. --- docs/en/prepare/index.mdx | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 4591a1f..7bc2702 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -66,7 +66,13 @@ Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, - On the traditional operating system layout, use `/cpaas/data/...`. - On Alauda OS nodes only `/var/cpaas` is writable, so use `/var/cpaas/data/...`. 4. Make sure the path survives node re-provisioning. -5. Create the directories the storage pods use and set their ownership. The examples use the traditional layout; on Alauda OS nodes replace `/cpaas` with `/var/cpaas`. +5. OpenSearch needs `vm.max_map_count` set to at least `262144`. The OpenSearch operator sets it from an init container, so nothing is required here unless your cluster enforces restricted Pod Security Admission, in which case that init container cannot run and you must set it on every node yourself: + + ```bash + sudo sysctl -w vm.max_map_count=262144 + echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf + ``` +6. Create the directories the storage pods use and set their ownership. The examples use the traditional layout; on Alauda OS nodes replace `/cpaas` with `/var/cpaas`. ```bash # ClickHouse runs as uid 101 @@ -82,7 +88,7 @@ Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, sudo chown -R 1000:1000 /cpaas/data/opensearch ``` -6. Decide how the volumes are provisioned: +7. Decide how the volumes are provisioned: | Approach | When to use it | What you must do | | --- | --- | --- | @@ -128,7 +134,7 @@ Save the YAML as `local-storage.yaml` and apply it: kubectl apply -f local-storage.yaml ``` -Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path`, and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example `/cpaas/data/clickhouse-0` and `/cpaas/data/clickhouse-1`, created with the ownership from step 5. +Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path`, and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example `/cpaas/data/clickhouse-0` and `/cpaas/data/clickhouse-1`, created with the ownership from step 6. ## Step 2: Install the Operators @@ -752,6 +758,7 @@ spec: type: simple config: auto.create.topics.enable: "false" + auto.leader.rebalance.enable: "false" default.replication.factor: "3" min.insync.replicas: "2" offsets.topic.replication.factor: "3" @@ -780,6 +787,7 @@ Do not omit these settings: | `message.max.bytes: "10485760"` | Audit batches are about 1.1–1.5 MiB each. The Kafka default of 1 MiB rejects every audit batch. | | `replica.fetch.max.bytes: "10485760"` | Must be at least `message.max.bytes`, otherwise replica synchronization stalls. | | `auto.create.topics.enable: "false"` | A mistyped topic name must not be created automatically and silently collect data. | +| `auto.leader.rebalance.enable: "false"` | Keeps the platform from moving partition leaders on its own. Rebalance deliberately during maintenance instead. | | `entityOperator.topicOperator` / `userOperator` | Without them, the `RdsTopic` and `RdsKafkaUser` resources in the next steps are not applied to the brokers. | ### 4.3 Wait for the broker cluster @@ -790,7 +798,15 @@ kubectl -n cpaas-system get rdsKafka cpaas-kafka \ kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka ``` -Wait until the `Ready` condition is `True` and all broker pods are `Running`. +Wait until the `Ready` condition is `True` and all broker pods are `Running`. The Kafka operator applies hard pod anti-affinity, so the three brokers must land on three different nodes: + +```bash +kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \ + -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +``` + +Three distinct node names are required. The same three nodes are also the minimum for the cluster to schedule at all. Confirm the required settings reached the brokers: @@ -1138,7 +1154,17 @@ The response must contain several tokens, for example `自然语言`, `处理`, ### 5.3 Create the account used by the logging components -The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. Replace `` with the password of the security plugin administrator and `` with a value of at least 32 characters. +The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. + +These calls authenticate as `admin`, the security plugin administrator. The operator generates that account and stores its credentials in the `-admin-password` Secret, so read the password from there and use it wherever `` appears below. `admin` / `admin` is the default, which is what the examples assume: + +```bash +kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.username}{" "}{.data.password}' | \ + while read u p; do echo "$(echo $u | base64 -d) / $(echo $p | base64 -d)"; done +``` + +Change that password before production and keep it in sync with the operator, or the operator's own health checks stop working; the platform OpenSearch knowledge base documents the procedure (write an `internal_users.yml` with the new hash to `securityconfig-secret`, create the matching `admin-credentials-secret`, then update the cluster). Replace `` with a value of at least 32 characters for the logging account. ```bash OS="https://" # the cluster service, for example https://cpaas-opensearch.cpaas-system.svc:9200 From 3a02e435d5dfdebafa8652a1b149c1ea124c1822 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 07:49:44 +0800 Subject: [PATCH 29/38] docs: fix ordering and wording in the OpenSearch account steps Reviewing the previous commit turned up three problems it introduced. The admin credential explanation was placed in 5.3, but 5.2 already uses `` in its analyze call, so the reader met the placeholder before the paragraph that explains it, and the paragraph claimed the placeholder appears "below" while it also appeared above. Move the explanation to the end of 5.1, where the cluster and its generated Secret exist, and point 5.3 back at it. Reading the credentials used a single jsonpath expression that concatenates two fields and pipes them through a read loop. Replace it with the two separate jsonpath reads already used for the ClickHouse admin password in 3.5, which is the form that has been exercised. Two wording fixes: the moved paragraph opened with "These calls" while nothing preceded it in 5.1, and the Kafka anti-affinity note ended with a sentence that said the same three nodes are the minimum to schedule rather than the actual constraint, which is that hard anti-affinity needs three schedulable nodes. --- docs/en/prepare/index.mdx | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 7bc2702..28aab6d 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -806,7 +806,7 @@ kubectl -n cpaas-system get pod \ -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' ``` -Three distinct node names are required. The same three nodes are also the minimum for the cluster to schedule at all. +Three distinct node names are required: hard anti-affinity needs at least three schedulable nodes. Confirm the required settings reached the brokers: @@ -1105,6 +1105,18 @@ kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ The health reports `unknown` and then `yellow` while the nodes start and the shards initialize. Wait for `green`. +The steps that follow authenticate as `admin`, the security plugin administrator. The operator generates that account and stores its credentials in the `-admin-password` Secret, so read the password from there and use it wherever `` appears in the steps below. `admin` / `admin` is the default, which is what the examples assume: + +```bash +OS_ADMIN_USER="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.username}' | base64 -d)" +OS_ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "$OS_ADMIN_USER / $OS_ADMIN_PASSWORD" +``` + +Change that password before production and keep it in sync with the operator, or the operator's own health checks stop working; the platform OpenSearch knowledge base documents the procedure (write an `internal_users.yml` with the new hash to `securityconfig-secret`, create the matching `admin-credentials-secret`, then update the cluster). Replace `` with a value of at least 32 characters for the logging account. + ### 5.2 Optional: install the Chinese analyzer plugin `analysis-ik` is optional. Without it the standard analyzer is used and Chinese text is not segmented; with it Chinese text is segmented. @@ -1154,17 +1166,7 @@ The response must contain several tokens, for example `自然语言`, `处理`, ### 5.3 Create the account used by the logging components -The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. - -These calls authenticate as `admin`, the security plugin administrator. The operator generates that account and stores its credentials in the `-admin-password` Secret, so read the password from there and use it wherever `` appears below. `admin` / `admin` is the default, which is what the examples assume: - -```bash -kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ - -o jsonpath='{.data.username}{" "}{.data.password}' | \ - while read u p; do echo "$(echo $u | base64 -d) / $(echo $p | base64 -d)"; done -``` - -Change that password before production and keep it in sync with the operator, or the operator's own health checks stop working; the platform OpenSearch knowledge base documents the procedure (write an `internal_users.yml` with the new hash to `securityconfig-secret`, create the matching `admin-credentials-secret`, then update the cluster). Replace `` with a value of at least 32 characters for the logging account. +The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. These calls authenticate as `admin`, using the password you read in 5.1. ```bash OS="https://" # the cluster service, for example https://cpaas-opensearch.cpaas-system.svc:9200 From ae7def9088706d001f5b89d8c3a2dadf802ac506 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 08:45:47 +0800 Subject: [PATCH 30/38] docs: reserve node-local volumes for their claims Two corrections against the version-matched knowledge base. Drop `auto.leader.rebalance.enable`. It came from the Kafka best practices article, which is written for ACP 3.14/3.15, and that article itself says to verify defaults against the operator actually installed. Nothing in the 4.x material asks for it. Rework the node-local storage steps around the ACP 4.3 procedure for Kafka on local disks, which lists the field failures this chapter was open to: a pod coming up with an empty or foreign log directory, and pods binding to each other's disks after an instance is recreated. The fix it prescribes is to reserve every PersistentVolume for one specific claim with `spec.claimRef`, and that is now what the chapter does: - Step 1 reserves each volume, adds the `project.cpaas.io/ALL_ALL` label the pvc-validator webhook requires in a project namespace, sets `allowVolumeExpansion: false`, and notes that `capacity.storage` is matching metadata rather than a quota. - Step 1 lists the claim-name shape per component, and gives Kafka its own directories per pod so the paths match the volumes that reference them. - Kafka's broker claim names contain a hash generated per instance, so the claim names cannot be known in advance. Creating the instance and then reserving the volumes for the claims it produced is now part of 4.2, before the wait-and-verify step. Verified on the cluster: the six Kafka claims went from Pending to bound one-to-one with their reserved volumes and all six pods started, and a three-node OpenSearch pool came up green with volumes reserved the same way. --- docs/en/prepare/index.mdx | 89 +++++++++++++++++++++++++++++++++------ 1 file changed, 77 insertions(+), 12 deletions(-) diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 28aab6d..f8d45d2 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -79,8 +79,8 @@ Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, sudo mkdir -p /cpaas/data/clickhouse sudo chown -R 101:101 /cpaas/data/clickhouse - # Kafka runs as uid 1001 - sudo mkdir -p /cpaas/data/kafka + # Kafka runs as uid 1001; one directory per pod, see below + sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-3 sudo chown -R 1001:1001 /cpaas/data/kafka # OpenSearch runs as uid 1000 @@ -95,16 +95,31 @@ Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, | Static local volumes | You are pinning each pod to a specific node, which is what the infra-node setup usually does | Create one StorageClass without a provisioner, and pre-create one PV per intended pod, each with `nodeAffinity` and `local.path` pointing at the directory above | | Dynamic provisioner | Your platform provides a block-storage provisioner | Create the StorageClass and let the claims bind dynamically; confirm the provisioner supports `ReadWriteOnce` block volumes and the throughput above | -For static local volumes, create one StorageClass and one PV per pod, then write the StorageClass name into `` in the manifests below. The number of PVs is the sum of the pods you plan: ClickHouse `shardsCount × replicasCount`, Kafka `replicas + controller.replicas`, and OpenSearch the sum of the node pool `replicas`. +For static local volumes, create one StorageClass and one PV per pod, and **reserve every PV for the claim it belongs to**. A `local` volume cannot follow its pod: unless a volume is reserved, a claim can bind a volume that was prepared for another pod or component, and after an instance is deleted and recreated its pods can bind each other's disks. Reserving the volume with `spec.claimRef` removes that risk, because the volume then only ever matches the claim named in it. + +Use a separate StorageClass per component. The example below creates the one for ClickHouse and reserves a single volume for the first ClickHouse replica. The number of PVs is the sum of the pods you plan: ClickHouse `shardsCount × replicasCount`, Kafka `replicas + controller.replicas`, and OpenSearch the sum of the node pool `replicas`. + +Claim names are deterministic for ClickHouse and OpenSearch, so their volumes can be reserved before the instance exists: + +| Component | Claim name for replica `` | +| --- | --- | +| ClickHouse | `data-volumeclaim-template-chi-----0` | +| OpenSearch | `data---` | +| Kafka | `data--broker--`, so read the names from the instance first — see Step 4 | ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: - name: cpaas-local + name: cpaas-local-clickhouse + labels: + # Required in a project namespace: without the grant, the pvc-validator + # admission webhook rejects every claim using this class. + project.cpaas.io/ALL_ALL: "true" provisioner: kubernetes.io/no-provisioner -reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false --- apiVersion: v1 kind: PersistentVolume @@ -113,10 +128,16 @@ metadata: spec: capacity: storage: 200Gi + volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain - storageClassName: cpaas-local + storageClassName: cpaas-local-clickhouse + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 local: path: /cpaas/data/clickhouse nodeAffinity: @@ -134,7 +155,9 @@ Save the YAML as `local-storage.yaml` and apply it: kubectl apply -f local-storage.yaml ``` -Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path`, and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example `/cpaas/data/clickhouse-0` and `/cpaas/data/clickhouse-1`, created with the ownership from step 6. +Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path` and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example `/cpaas/data/clickhouse-0` and `/cpaas/data/clickhouse-1`, created with the ownership from step 6. + +`capacity.storage` on a `local` volume is matching metadata, not a quota: nothing stops a pod from filling the underlying disk past it. Set it to the real usable size and enforce retention on the storage side as well. ## Step 2: Install the Operators @@ -758,7 +781,6 @@ spec: type: simple config: auto.create.topics.enable: "false" - auto.leader.rebalance.enable: "false" default.replication.factor: "3" min.insync.replicas: "2" offsets.topic.replication.factor: "3" @@ -780,6 +802,50 @@ Save the YAML as `cpaas-kafka.yaml` and apply it: kubectl apply -f cpaas-kafka.yaml ``` +The instance creates its claims immediately. They stay `Pending` until you reserve volumes for them, because the broker claim names contain a hash that is generated per instance, which is why the volumes cannot be prepared up front: + +```bash +kubectl -n cpaas-system get pvc \ + -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' +``` + +Copy the six names exactly — three `...-broker-...` and three `...-controller-...` — and create one pre-bound volume per claim. Set `capacity.storage` to the size the claim asks for, point `local.path` at that pod's directory, and pin the node: + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-kafka-broker-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-kafka + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-cpaas-kafka-broker--0 + local: + path: /cpaas/data/kafka/broker-0 + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +Save the YAML as `cpaas-kafka-volumes.yaml`, repeat the `PersistentVolume` part for all six claims, and apply it. The claims bind as soon as their volumes exist, and the brokers and controllers start: + +```bash +kubectl apply -f cpaas-kafka-volumes.yaml +``` + Do not omit these settings: | Setting | Why it is required | @@ -787,7 +853,6 @@ Do not omit these settings: | `message.max.bytes: "10485760"` | Audit batches are about 1.1–1.5 MiB each. The Kafka default of 1 MiB rejects every audit batch. | | `replica.fetch.max.bytes: "10485760"` | Must be at least `message.max.bytes`, otherwise replica synchronization stalls. | | `auto.create.topics.enable: "false"` | A mistyped topic name must not be created automatically and silently collect data. | -| `auto.leader.rebalance.enable: "false"` | Keeps the platform from moving partition leaders on its own. Rebalance deliberately during maintenance instead. | | `entityOperator.topicOperator` / `userOperator` | Without them, the `RdsTopic` and `RdsKafkaUser` resources in the next steps are not applied to the brokers. | ### 4.3 Wait for the broker cluster @@ -802,7 +867,7 @@ Wait until the `Ready` condition is `True` and all broker pods are `Running`. Th ```bash kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' ``` @@ -812,7 +877,7 @@ Confirm the required settings reached the brokers: ```bash KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ -o jsonpath='{.items[0].metadata.name}')" kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ @@ -968,7 +1033,7 @@ Build the client properties inside the broker pod from the user Secret. The brok ```bash KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ -o jsonpath='{.items[0].metadata.name}')" kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ From e95098053e3c6765fe88d7c5f186b7979f08b469 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 08:49:47 +0800 Subject: [PATCH 31/38] docs: move the external storage install path into the installation chapter Installing the log storage against a storage or Kafka service you provide belongs in the installation chapter, not in the upgrade flow. The upgrade chapter was carrying the entire manifest set for it: the two connection Secrets, the PlatformLogForward with all its comments, and the notes about shard and replica counts. Add that procedure to the installation chapter as Install Log Storage with External ClickHouse or OpenSearch, covering the OpenSearch and ClickHouse Secret variants, both PlatformLogForward variants, and the readiness check. The upgrade chapter now links to it and documents only what makes the two different: the log.alauda.io/legacy-es-upgrade annotation and the path it selects, which waits for the queued legacy data instead of switching the log entry point directly. Its steps renumber from six to five, and the flow table matches the new numbering. --- docs/en/install_log.mdx | 163 +++++++++++++++++++++++++++++++++++++ docs/en/upgrade/index.mdx | 166 +++++--------------------------------- 2 files changed, 185 insertions(+), 144 deletions(-) diff --git a/docs/en/install_log.mdx b/docs/en/install_log.mdx index 783fdaa..407c63e 100644 --- a/docs/en/install_log.mdx +++ b/docs/en/install_log.mdx @@ -612,6 +612,169 @@ config: Before applying these scheduling rules, make sure your infra node planning and local storage placement are compatible. For the node planning considerations, see [Planning Infra Nodes for Logging Storage](./how_to/infra_nodes.mdx). +## Install Log Storage with External ClickHouse or OpenSearch + +Use this path when the log, event, and audit data must live in storage you operate — a ClickHouse or an OpenSearch cluster, plus a Kafka service — instead of the storage that the platform storage plugins install. OpenSearch as a log storage target is only available through this path. + +### Prepare the storage first + +Create the target cluster and the Kafka service, including their accounts, ACLs, and topics, and record the connection details: + +- [Environment Preparation](../prepare/index.mdx) + +Those recorded values are the inputs to the Secrets below. + +### Step 1: Create the connection Secrets + +**Run on the workload cluster.** + +Create the Secret for your target storage and the Secret for the Kafka service in `cpaas-system`. Create only the storage Secret that matches your target. + +#### Target OpenSearch + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first + username: "" # Can be omitted when the target allows anonymous access + password: "" # Can be omitted when the target allows anonymous access + tls.ca: |- # Required when the target uses HTTPS with a private CA, so the platform can verify the target + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +--- +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + bootstrap: "" # Required; Kafka addresses in host:port form, separated by commas + kafkaClusterName: "" # Required; Kafka broker resource name; must match the actual name + username: "" # Required; Kafka user name + password: "" # Required; at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts + sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 + topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC + topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC + topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC + tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +#### Target ClickHouse + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoint: "https://:8443" # Required; ClickHouse address, including protocol and port + cluster: "replicated" # Must match the ClickHouse cluster name; ACP baseline default is replicated + database: "observability" # Target database name, defaults to observability + username: "" # ClickHouse user name + password: "" # ClickHouse password + tls.ca: |- # Required when the target uses HTTPS with a private CA + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +`endpoints` accepts multiple comma-separated HTTP(S) URLs. Some data paths use only the first address, so put the highly available load balancer or coordinator endpoint first, not a single data node, and do not add leading whitespace before the first URL. + +Do not set `tls.insecure_skip_verify: "true"`. Provide `tls.ca` instead, so the platform can verify the target. + +### Step 2: Create the PlatformLogForward + +**Run on the workload cluster.** + +Create one `PlatformLogForward` for your target. + +#### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + externalStorage: + type: opensearch # Target storage type + secretRef: + name: platform-default-os-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +#### Target ClickHouse + +The `output.type` field is required for this target. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + output: + type: clickhouse # Required when the target is ClickHouse + externalStorage: + type: clickhouse # Target storage type + shards: 1 # Actual shard count of the target ClickHouse + replicas: 1 # Actual replica count of the target ClickHouse + secretRef: + name: platform-default-ch-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. + +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. + +Save the YAML as `platform-log-forward.yaml` and apply it: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +### Step 3: Verify + +Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column reaches `Ready` and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +Then produce or locate new log, event, and audit records and confirm that you can query them from the target storage. + ## Install Alauda Container Platform Log Collector Plugin ### Console diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 795d11c..fa9e3dd 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -44,7 +44,7 @@ Before you start, ensure that: - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches fall back to a standard analyzer without Chinese word segmentation: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - - The target storage and Kafka connection details prepared according to their product documentation. + - The target storage and Kafka connection details prepared as described in [Environment Preparation](../prepare/index.mdx). 4. If you migrate historical data, you have the migration image provided for this release, with the complete registry, tag, or digest. Do not reuse an image from an earlier version. 5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. 6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. @@ -56,7 +56,7 @@ During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafk ## Target Storage Preparation -The operator does not create the external OpenSearch/ClickHouse cluster or the new Kafka service. Create them separately, then provide the connection details through the Secrets in Step 1. +The operator does not create the external OpenSearch/ClickHouse cluster or the new Kafka service. Create them as described in [Environment Preparation](../prepare/index.mdx), then provide the connection details through the Secrets in Step 1. Plan the size with [Log Component Capacity Planning](../architecture/capacity_planning.mdx), and do not plan below the current Alauda Container Platform Log Storage for Elasticsearch deployment. The target must hold the migrated history plus new traffic. The reference disk configuration in that guide is `6000 IOPS` and `250 MB/s` read/write on dedicated SSD mounts; if the actual storage is weaker, move to a larger profile. @@ -99,154 +99,34 @@ Use this order. Each gate must pass before the next step starts. | Step | Where | Action | Gate to continue | | --- | --- | --- | --- | -| 1 | Workload cluster | Prepare the target storage, Kafka, and connection Secrets | The target storage and topics are reachable | -| 2 | Workload cluster | Create `PlatformLogForward` | `Phase=Ready` and `LegacyESUpgradeCompleted` | -| 3 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | -| 4 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | -| 5 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | -| 6 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | +| 1 | Workload cluster | Install the new data path: connection Secrets plus `PlatformLogForward` with the upgrade annotation | `Phase=Ready` and `LegacyESUpgradeCompleted` | +| 2 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | +| 3 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | +| 4 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | +| 5 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | ## Upgrade Procedure -### Step 1: Prepare the target storage, Kafka, and connection Secrets +### Step 1: Install the new data path with the upgrade annotation -**Run on the workload cluster.** - -Create two Secrets in `cpaas-system` on the workload cluster: one for the target storage, and one for the new Kafka service. Do not overwrite or reuse the Secrets of the legacy Elasticsearch and Kafka. - -#### Target OpenSearch - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first - username: "" # Can be omitted when the target allows anonymous access - password: "" # Can be omitted when the target allows anonymous access - tls.ca: |- # Required when you migrate historical data and the target uses HTTPS with a private CA, so the migration can verify the target before writing - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- ---- -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - bootstrap: "" # Required; Kafka addresses in host:port form, separated by commas - kafkaClusterName: "" # Required; Kafka broker resource name; must match the actual name - username: "" # Required; Kafka user name - password: "" # Required; at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts - sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 - topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC - topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC - topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC - tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -`endpoints` accepts multiple comma-separated HTTP(S) URLs. Some data paths use only the first address, so put the highly available load balancer or coordinator endpoint first, not a single data node, and do not add leading whitespace before the first URL. - -#### Target ClickHouse - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoint: "https://:8443" # Required; ClickHouse address, including protocol and port - cluster: "replicated" # Must match the ClickHouse cluster name; ACP baseline default is replicated - database: "observability" # Target database name, defaults to observability - username: "" # ClickHouse user name - password: "" # ClickHouse password - tls.ca: |- # Required when the target uses HTTPS with a private CA - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -The new Kafka service uses the same `platform-default-mq-conn` as in the OpenSearch section. - -When you migrate historical data, do not set `tls.insecure_skip_verify: "true"` in the target storage connection Secret; provide `tls.ca` instead so the migration can verify the target. - -### Step 2: Create the new data path - -**Run on the workload cluster.** - -Create one `PlatformLogForward` for your target. It must use `installMode: Fresh` and the `log.alauda.io/legacy-es-upgrade: "true"` annotation. Do not use `installMode: Adopt`. +The new data path is installed the same way as a fresh installation: create the connection Secrets for the target storage and for Kafka, then create one `PlatformLogForward`. Follow **Install Log Storage with External ClickHouse or OpenSearch** in [Installation](../install_log.mdx) for those manifests and the notes that come with them. -#### Target OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # Fixed cluster singleton name, do not change - annotations: - log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow -spec: - installMode: Fresh # Always Fresh, do not change it to Adopt - externalStorage: - type: opensearch # Target storage type - secretRef: - name: platform-default-os-conn # Target storage connection Secret created in Step 1 - namespace: cpaas-system - externalMessageQueue: - type: kafka # Message queue type, currently only kafka - secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret - namespace: cpaas-system -``` - -#### Target ClickHouse - -The `output.type` field is required for this target. +**The annotation is the only difference.** Add `log.alauda.io/legacy-es-upgrade: "true"` to the `PlatformLogForward`, so the platform enters the legacy Elasticsearch upgrade flow instead of switching the log entry point directly: ```yaml apiVersion: log.alauda.io/v1alpha1 kind: PlatformLogForward metadata: - name: platform-default # Fixed cluster singleton name, do not change + name: platform-default # Fixed cluster singleton name, do not change annotations: - log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow + log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow spec: - installMode: Fresh # Always Fresh, do not change it to Adopt - output: - type: clickhouse # Required when the target is ClickHouse - externalStorage: - type: clickhouse # Target storage type - shards: 1 # Actual shard count of the target ClickHouse - replicas: 1 # Actual replica count of the target ClickHouse - secretRef: - name: platform-default-ch-conn # Target storage connection Secret created in Step 1 - namespace: cpaas-system - externalMessageQueue: - type: kafka # Message queue type, currently only kafka - secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret - namespace: cpaas-system + installMode: Fresh # Always Fresh, do not change it to Adopt + # externalStorage and externalMessageQueue: as in Installation, pointing at the + # target storage and Kafka connection Secrets you created above ``` -`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. - -`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. - -Save the YAML as `platform-log-forward.yaml` and apply it: - -```bash -kubectl apply -f platform-log-forward.yaml -``` +Keep `installMode: Fresh`; `Adopt` is not part of this flow. The new data path is not available immediately. The platform first creates it, then switches the log entry point, and finally waits for the data that queued up in the old cluster to be consumed; the time this takes depends on the backlog. Watch the status until it finishes, and press `Ctrl+C` to stop: @@ -254,9 +134,7 @@ The new data path is not available immediately. The platform first creates it, t kubectl get platformlogforward platform-default -w ``` -The `Phase` column reaches `Ready`, and the `Ready` column becomes `True`. Continue only after you see `Ready`. - -To follow the progress or troubleshoot, read the status conditions: +The `Phase` column reaches `Ready`, and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: ```bash kubectl get platformlogforward platform-default \ @@ -269,7 +147,7 @@ Once this step is complete, produce or locate new log, event, and audit records, If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform drains the queued data through the legacy path automatically and records `LegacyKafkaDrained` when the old consumer lag reaches zero; `LegacyESUpgradeCompleted` is the gate for this procedure. -### Step 3: Prepare the historical data migration (optional) +### Step 2: Prepare the historical data migration (optional) **Run on the workload cluster.** @@ -365,13 +243,13 @@ kubectl -n cpaas-system get legacyesmigration platform-es-history \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` -In this flow, data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 4 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. +In this flow, data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 3 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. -### Step 4: Uninstall the legacy Elasticsearch storage plugin +### Step 3: Uninstall the legacy Elasticsearch storage plugin :::warning Run this step only when all of these conditions are true: @@ -481,7 +359,7 @@ If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and r After the stability check passes, produce new log, event, and audit records and confirm that they can be queried from the new target. Do not continue to migration verification if the new data path is unhealthy. -### Step 5: Complete and verify the historical migration (optional) +### Step 4: Complete and verify the historical migration (optional) **Run on the workload cluster.** @@ -507,7 +385,7 @@ If the phase is `Blocked` or `Failed`, do not delete the migration resource, Job If no migration was created, record the explicit decision that historical data is not required, and keep the protected source volumes until that decision is approved. -### Step 6: Retained volumes +### Step 5: Retained volumes The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until migration and target validation are complete. Do not remove protection annotations, delete PVCs or PVs, or remove finalizers. To release the volumes after validation, contact Alauda support or follow the separate approved cleanup procedure. From e8dad3d959c788a33ea1f211c93ccfe8fd2c31fe Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 09:47:06 +0800 Subject: [PATCH 32/38] docs: give the external storage install its own chapter and fix the ClickHouse account Move the external storage install procedure out of install_log.mdx into a new Installation chapter. The older page covers installing the platform storage plugins and is not where this path belongs; the new page takes the procedure that the upgrade chapter used to carry, without the upgrade annotation. Create the ClickHouse account the way the product always has. Running the whole flow end to end showed the SQL user the chapter created needed four grants it was never given: SELECT on system.clusters for razor's readiness probe, SELECT on system.zookeeper for the keeper probe, CLUSTER for ON CLUSTER DDL, and TABLE ENGINE ON Distributed for the distributed table the migrations create. The product's own ClickHouse chart instead declares the account in the ClickHouseInstallation users section with GRANT ALL ON *.* WITH GRANT OPTION, which carries all of those without a grant statement anywhere. This chapter now does the same: 3.1 creates the two password Secrets, the manifests declare admin and platform-logging, and 3.5 just reads the password and verifies it. Verified end to end against ClickHouse and Kafka prepared by this chapter: the account runs all four operations with no grants at all, the PlatformLogForward reaches Ready, schema-migrations completes, and 19 tables exist in observability. --- docs/en/install/index.mdx | 170 ++++++++++++++++++++++++++++++++++++++ docs/en/install_log.mdx | 163 ------------------------------------ docs/en/prepare/index.mdx | 44 +++++++--- 3 files changed, 201 insertions(+), 176 deletions(-) create mode 100644 docs/en/install/index.mdx diff --git a/docs/en/install/index.mdx b/docs/en/install/index.mdx new file mode 100644 index 0000000..51a2610 --- /dev/null +++ b/docs/en/install/index.mdx @@ -0,0 +1,170 @@ +--- +weight: 14 +--- + +# Installation + +This chapter installs the logging components against the storage and message queue that you operate: a ClickHouse or OpenSearch cluster and a Kafka service. Prepare them first, including their accounts, ACLs, and topics: + +- [Environment Preparation](../prepare/index.mdx) + +The connection details you recorded there are the inputs to the Secrets below. + +Use this path when the log, event, and audit data must live in storage you operate — a ClickHouse or an OpenSearch cluster, plus a Kafka service — instead of the storage that the platform storage plugins install. OpenSearch as a log storage target is only available through this path. + +## Prepare the storage first + +Create the target cluster and the Kafka service, including their accounts, ACLs, and topics, and record the connection details: + +- [Environment Preparation](../prepare/index.mdx) + +Those recorded values are the inputs to the Secrets below. + +## Step 1: Create the connection Secrets + +**Run on the workload cluster.** + +Create the Secret for your target storage and the Secret for the Kafka service in `cpaas-system`. Create only the storage Secret that matches your target. + +### Target OpenSearch + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first + username: "" # Can be omitted when the target allows anonymous access + password: "" # Can be omitted when the target allows anonymous access +--- +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + bootstrap: "" # Required; Kafka addresses in host:port form, separated by commas + kafkaClusterName: "" # Required; Kafka broker resource name; must match the actual name + username: "" # Required; Kafka user name + password: "" # Required; at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts + sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 + topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC + topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC + topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC + tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +### Target ClickHouse + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later + namespace: cpaas-system +type: Opaque +stringData: + endpoint: "https://:8443" # Required; ClickHouse address, including protocol and port + cluster: "replicated" # Must match the ClickHouse cluster name; ACP baseline default is replicated + database: "observability" # Target database name, defaults to observability + username: "" # ClickHouse user name + password: "" # ClickHouse password + tls.ca: |- # Required when the target uses HTTPS with a private CA + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +`endpoints` accepts multiple comma-separated HTTP(S) URLs. Some data paths use only the first address, so put the highly available load balancer or coordinator endpoint first, not a single data node, and do not add leading whitespace before the first URL. + +The OpenSearch Secret carries no TLS key. The platform connects to OpenSearch with certificate verification disabled, so an endpoint behind a private CA needs no entry here; use `https://` in `endpoints` and the connection is established regardless of the issuer. + +For ClickHouse, do not set `tls.insecure_skip_verify: "true"`; provide `tls.ca` instead so the platform can verify the target. + +## Step 2: Create the PlatformLogForward + +**Run on the workload cluster.** + +Create one `PlatformLogForward` for your target. + +### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + externalStorage: + type: opensearch # Target storage type + secretRef: + name: platform-default-os-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +### Target ClickHouse + +The `output.type` field is required for this target. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + output: + type: clickhouse # Required when the target is ClickHouse + externalStorage: + type: clickhouse # Target storage type + shards: 1 # Actual shard count of the target ClickHouse + replicas: 1 # Actual replica count of the target ClickHouse + secretRef: + name: platform-default-ch-conn # Target storage connection Secret created in Step 1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + namespace: cpaas-system +``` + +`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. + +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. + +Save the YAML as `platform-log-forward.yaml` and apply it: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +## Step 3: Verify + +Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column reaches `Ready` and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +Then produce or locate new log, event, and audit records and confirm that you can query them from the target storage. diff --git a/docs/en/install_log.mdx b/docs/en/install_log.mdx index 407c63e..783fdaa 100644 --- a/docs/en/install_log.mdx +++ b/docs/en/install_log.mdx @@ -612,169 +612,6 @@ config: Before applying these scheduling rules, make sure your infra node planning and local storage placement are compatible. For the node planning considerations, see [Planning Infra Nodes for Logging Storage](./how_to/infra_nodes.mdx). -## Install Log Storage with External ClickHouse or OpenSearch - -Use this path when the log, event, and audit data must live in storage you operate — a ClickHouse or an OpenSearch cluster, plus a Kafka service — instead of the storage that the platform storage plugins install. OpenSearch as a log storage target is only available through this path. - -### Prepare the storage first - -Create the target cluster and the Kafka service, including their accounts, ACLs, and topics, and record the connection details: - -- [Environment Preparation](../prepare/index.mdx) - -Those recorded values are the inputs to the Secrets below. - -### Step 1: Create the connection Secrets - -**Run on the workload cluster.** - -Create the Secret for your target storage and the Secret for the Kafka service in `cpaas-system`. Create only the storage Secret that matches your target. - -#### Target OpenSearch - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first - username: "" # Can be omitted when the target allows anonymous access - password: "" # Can be omitted when the target allows anonymous access - tls.ca: |- # Required when the target uses HTTPS with a private CA, so the platform can verify the target - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- ---- -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - bootstrap: "" # Required; Kafka addresses in host:port form, separated by commas - kafkaClusterName: "" # Required; Kafka broker resource name; must match the actual name - username: "" # Required; Kafka user name - password: "" # Required; at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts - sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 - topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC - topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC - topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC - tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -#### Target ClickHouse - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoint: "https://:8443" # Required; ClickHouse address, including protocol and port - cluster: "replicated" # Must match the ClickHouse cluster name; ACP baseline default is replicated - database: "observability" # Target database name, defaults to observability - username: "" # ClickHouse user name - password: "" # ClickHouse password - tls.ca: |- # Required when the target uses HTTPS with a private CA - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- -``` - -`endpoints` accepts multiple comma-separated HTTP(S) URLs. Some data paths use only the first address, so put the highly available load balancer or coordinator endpoint first, not a single data node, and do not add leading whitespace before the first URL. - -Do not set `tls.insecure_skip_verify: "true"`. Provide `tls.ca` instead, so the platform can verify the target. - -### Step 2: Create the PlatformLogForward - -**Run on the workload cluster.** - -Create one `PlatformLogForward` for your target. - -#### Target OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # Fixed cluster singleton name, do not change -spec: - installMode: Fresh # Always Fresh, do not change it to Adopt - externalStorage: - type: opensearch # Target storage type - secretRef: - name: platform-default-os-conn # Target storage connection Secret created in Step 1 - namespace: cpaas-system - externalMessageQueue: - type: kafka # Message queue type, currently only kafka - secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret - namespace: cpaas-system -``` - -#### Target ClickHouse - -The `output.type` field is required for this target. - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: PlatformLogForward -metadata: - name: platform-default # Fixed cluster singleton name, do not change -spec: - installMode: Fresh # Always Fresh, do not change it to Adopt - output: - type: clickhouse # Required when the target is ClickHouse - externalStorage: - type: clickhouse # Target storage type - shards: 1 # Actual shard count of the target ClickHouse - replicas: 1 # Actual replica count of the target ClickHouse - secretRef: - name: platform-default-ch-conn # Target storage connection Secret created in Step 1 - namespace: cpaas-system - externalMessageQueue: - type: kafka # Message queue type, currently only kafka - secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret - namespace: cpaas-system -``` - -`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. - -`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. - -Save the YAML as `platform-log-forward.yaml` and apply it: - -```bash -kubectl apply -f platform-log-forward.yaml -``` - -### Step 3: Verify - -Watch the status until it finishes, and press `Ctrl+C` to stop: - -```bash -kubectl get platformlogforward platform-default -w -``` - -The `Phase` column reaches `Ready` and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: - -```bash -kubectl get platformlogforward platform-default \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' -``` - -Then produce or locate new log, event, and audit records and confirm that you can query them from the target storage. - ## Install Alauda Container Platform Log Collector Plugin ### Console diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index f8d45d2..4d59a34 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -204,11 +204,15 @@ An operator that does not watch `cpaas-system` ignores the resources below silen Skip this step when the target is OpenSearch. -### 3.1 Create the admin password Secret +### 3.1 Create the password Secrets + +The instance defines two accounts: `admin` for administration, and `platform-logging` for the logging components. Both read their password from a Secret, so create both before you create the instance. ```bash kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ --from-literal=password="$(openssl rand -hex 16)" +kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" ``` ### 3.2 Create the Keeper client Service (three nodes and above) @@ -270,6 +274,17 @@ spec: - "::/0" admin/grants/query: - GRANT ALL ON *.* WITH GRANT OPTION + # The account the logging components use. It is declared here, like the admin + # account, so it carries the privileges the components need without a separate + # set of GRANT statements. + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION profiles: default/allow_nondeterministic_mutations: "1" @@ -414,6 +429,17 @@ spec: - "::/0" admin/grants/query: - GRANT ALL ON *.* WITH GRANT OPTION + # The account the logging components use. It is declared here, like the admin + # account, so it carries the privileges the components need without a separate + # set of GRANT statements. + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION profiles: default/allow_nondeterministic_mutations: "1" @@ -654,27 +680,19 @@ kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ # expect: cpaas-clickhouse-keeper 9181 ``` -### 3.5 Create the account used by the logging components +### 3.5 Read the logging account password -The account must be able to create and alter tables in the target database. Scope it to that database. +The account is declared in the `ClickHouseInstallation` you applied in 3.3, so there is nothing to create here and no `GRANT` statements to run: like the `admin` account, it is a server-configuration user and carries the privileges the logging components need. -Run these on the host. They generate the password, then run the SQL inside the ClickHouse pod. The account name needs backticks because it contains a hyphen, and `SYSTEM DROP DNS CACHE` is required by the retention cleaner. Record the printed password: the connection details in Step 6 need it, and 3.6 reuses the `LOG_PASSWORD` variable. +Read its password for the connection details in Step 6. 3.6 reuses the `CH_POD` and `LOG_PASSWORD` variables, so run both in the same shell. ```bash CH_POD="$(kubectl -n cpaas-system get pod \ -l clickhouse.altinity.com/chi=cpaas-clickhouse \ -o jsonpath='{.items[0].metadata.name}')" -ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-basic-auth \ +LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ -o jsonpath='{.data.password}' | base64 -d)" -LOG_PASSWORD="$(openssl rand -hex 16)" echo "platform-logging password: $LOG_PASSWORD" - -kubectl -n cpaas-system exec -i "$CH_POD" -- clickhouse-client \ - --user admin --password "$ADMIN_PASSWORD" --multiquery < Date: Thu, 17 Sep 2026 09:52:17 +0800 Subject: [PATCH 33/38] docs(zh): add the Chinese installation and preparation chapters Mirror the two chapters that were just reworked, so the Chinese tree matches the English one: - docs/zh/install/index.mdx, the new external storage installation chapter, without the upgrade annotation and with the same OpenSearch Secret shape as the English page. - docs/zh/prepare/index.mdx, including the ClickHouse account declared in the ClickHouseInstallation with GRANT ALL ON *.* instead of a SQL user with grant statements. Checked both pairs: the code blocks are identical after stripping comments, the heading ranks match, every bash block passes bash -n, every YAML block parses, and each sourceSHA matches its English source. --- docs/zh/install/index.mdx | 165 +++++ docs/zh/prepare/index.mdx | 1348 +++++++++++++++++++++++++++++++++++++ 2 files changed, 1513 insertions(+) create mode 100644 docs/zh/install/index.mdx create mode 100644 docs/zh/prepare/index.mdx diff --git a/docs/zh/install/index.mdx b/docs/zh/install/index.mdx new file mode 100644 index 0000000..90208e3 --- /dev/null +++ b/docs/zh/install/index.mdx @@ -0,0 +1,165 @@ +--- +weight: 14 +sourceSHA: f09d3facc79a57b4271628a04fea25c43cf9d09f43d5a42269d32c096c78b3d6 +--- + +# 安装 + +本章把日志组件安装到由你自行运维的存储与消息队列上：一套 ClickHouse 或 OpenSearch 集群，以及一个 Kafka 服务。请先完成它们的准备，包括账号、ACL 与 topic： + +## 先准备存储 + +请先创建目标集群与 Kafka 服务，包括它们的账号、ACL 与 topic，并记录连接信息： + +- [环境准备](https://docs.alauda.cn/logging-service/4.3/prepare/index.html) + +在那里记录下来的连接信息，就是下面这些 Secret 的输入。 + +## 步骤 1：创建连接 Secret + +**在运行日志组件的集群上执行。** + +在 `cpaas-system` 中创建目标存储的 Secret 和 Kafka 服务的 Secret。请只创建与你的目标类型对应的那个存储 Secret。 + +### 目标 OpenSearch + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 + namespace: cpaas-system +type: Opaque +stringData: + endpoints: "https://:9200" # 必填；逗号分隔的 HTTP(S) 地址；请把高可用协调节点或负载均衡地址放在最前面 + username: "" # 目标端允许匿名访问时可省略 + password: "" # 目标端允许匿名访问时可省略 +--- +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn # Kafka 连接 Secret 名称，创建 PlatformLogForward 时引用 + namespace: cpaas-system +type: Opaque +stringData: + bootstrap: "" # 必填；Kafka 地址，host:port 形式，逗号分隔 + kafkaClusterName: "" # 必填；Kafka broker 资源名称，必须与实际名称一致 + username: "" # 必填；Kafka 用户名 + password: "" # 必填；在 Alauda OS 节点或其他启用 FIPS 的主机上至少 32 个字符 + sasl_mechanism: "SCRAM-SHA-512" # 可选，默认 SCRAM-SHA-512 + topics.log: "ALAUDA_LOG_TOPIC" # 可选，日志 topic 名，默认 ALAUDA_LOG_TOPIC + topics.event: "ALAUDA_EVENT_TOPIC" # 可选，事件 topic 名，默认 ALAUDA_EVENT_TOPIC + topics.audit: "ALAUDA_AUDIT_TOPIC" # 可选，审计 topic 名，默认 ALAUDA_AUDIT_TOPIC + tls.ca: |- # Kafka 使用 TLS 且证书不被系统信任时必填 + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +### 目标 ClickHouse + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 + namespace: cpaas-system +type: Opaque +stringData: + endpoint: "https://:8443" # 必填；ClickHouse 地址，包含协议和端口 + cluster: "replicated" # 必须与 ClickHouse 集群名一致；ACP 基线默认为 replicated + database: "observability" # 目标数据库名，默认 observability + username: "" # ClickHouse 用户名 + password: "" # ClickHouse 密码 + tls.ca: |- # 目标端使用 HTTPS 私有 CA 时必填 + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +`endpoints` 支持以逗号分隔的多个 HTTP(S) 地址。部分数据链路只使用第一个地址，因此请把高可用的负载均衡器或 coordinator 地址放在最前面，不要使用单个数据节点，也不要在第一个 URL 前留空格。 + +OpenSearch 的 Secret 不包含 TLS 相关字段。平台连接 OpenSearch 时关闭了证书校验，因此使用私有 CA 的地址不需要在这里填写任何证书;地址使用 `https://` 即可建立连接，与签发方无关。 + +ClickHouse 请勿设置 `tls.insecure_skip_verify: "true"`，而应提供 `tls.ca`，以便平台校验目标端。 + +## 步骤 2：创建 PlatformLogForward + +**在运行日志组件的集群上执行。** + +为你的目标类型创建一个 `PlatformLogForward`。 + +### 目标 OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 固定的集群单例名称，请勿修改 +spec: + installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + externalStorage: + type: opensearch # 目标存储类型 + secretRef: + name: platform-default-os-conn # 步骤 1 创建的目标存储连接 Secret + namespace: cpaas-system + externalMessageQueue: + type: kafka # 消息队列类型，目前仅支持 kafka + secretRef: + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret + namespace: cpaas-system +``` + +### 目标 ClickHouse + +目标为 ClickHouse 时必须填写 `output.type` 字段。 + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 固定的集群单例名称，请勿修改 +spec: + installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + output: + type: clickhouse # 目标为 ClickHouse 时必填 + externalStorage: + type: clickhouse # 目标存储类型 + shards: 1 # 目标 ClickHouse 的实际分片数 + replicas: 1 # 目标 ClickHouse 的实际副本数 + secretRef: + name: platform-default-ch-conn # 步骤 1 创建的目标存储连接 Secret + namespace: cpaas-system + externalMessageQueue: + type: kafka # 消息队列类型，目前仅支持 kafka + secretRef: + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret + namespace: cpaas-system +``` + +`externalStorage.shards` 与 `externalStorage.replicas` 必须与 ClickHouse 实际拓扑一致。两者默认都是 `1`;多分片或多副本部署中填错会导致部分拓扑不被使用。 + +`PlatformLogForward` 是集群级资源，请不要为它添加 `metadata.namespace`;`secretRef` 里的 `namespace` 字段仍用于指向 `cpaas-system` 中的连接 Secret。CRD 默认值为 `aggregateVector.replicas: 3` 和 `razor.replicas: 2`;如果容量或调度规划需要不同的副本数，请显式设置。 + +请把 YAML 保存为 `platform-log-forward.yaml` 并应用： + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +## 步骤 3：校验 + +观察状态直到完成，按 `Ctrl+C` 停止： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 列会变为 `Ready`，`Ready` 列会变为 `True`。需要跟踪进度或排查问题时，读取 status conditions： + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +随后产生或找到新的日志、事件与审计记录，确认可以从目标存储中查询到它们。 diff --git a/docs/zh/prepare/index.mdx b/docs/zh/prepare/index.mdx new file mode 100644 index 0000000..f089c03 --- /dev/null +++ b/docs/zh/prepare/index.mdx @@ -0,0 +1,1348 @@ +--- +weight: 13 +sourceSHA: 78020a1b837c09abfd42a624d3d6a1fc12b64ac26c3152c44d72f724197548e8 +--- + +# 环境准备 + +本文说明如何搭建日志组件使用的 ClickHouse 或 OpenSearch 3.7.0 集群和 Kafka 服务，包括节点与磁盘、Operator、集群及其账号，以及 Kafka 的密码、用户、ACL 和 topic。 + +请按顺序执行并完成校验。 + +## 开始之前 + +请先确认： + +1. 你拥有运行日志组件的集群的管理员权限。 +2. 已按步骤 1 规划好节点和磁盘。 +3. 平台市场中已上架以下 Operator 包：`clickhouse-operator`、Alauda Kafka Operator、`opensearch-operator`。 + +请结合[日志组件容量规划](https://docs.alauda.cn/logging-service/4.3/architecture/capacity_planning.html)和下面的表格确定规格，并按[为日志存储规划基础设施节点](https://docs.alauda.cn/logging-service/4.3/how_to/infra_nodes.html)把工作负载放到独占节点上。 + +本章所有命令都请在能访问该集群的 `kubectl` 主机上执行。每段 YAML 都需要先保存成文件再用 `kubectl apply -f` 应用；每段代码块下方都给出了文件名和对应的 apply 命令。 + +## 步骤 0：选择目标类型和规格 + +先确定目标类型（ClickHouse 或 OpenSearch）和规格。 + +### ClickHouse 规格 + +CPU 和内存是每个 ClickHouse Pod 的容器 limit。 + +| 规格 | ClickHouse Pod 数 | 拓扑 | 每 Pod CPU limit | 每 Pod 内存 limit | 实测吞吐 | +| --- | --- | --- | --- | --- | --- | +| 单节点 | 1 | 1 分片 × 1 副本 | 2C | 4G | 18,000 logs/s | +| 三节点 | 3 | 1 分片 × 3 副本 | 2C | 4G | 20,000 logs/s | +| 六节点 | 6 | 2 分片 × 3 副本 | 4C | 8G | 40,000 logs/s | +| 九节点 | 9 | 3 分片 × 3 副本 | 4C | 8G | 69,000 logs/s | + +单节点规格仅用于验证环境。生产环境请从三节点规格起步；单个分片无法容纳数据时，再扩展到六节点或九节点。 + +### Kafka + +请准备三个 broker，每个 limit 为 2C/4G，另外还有步骤 4 清单中三个 limit 为 1C/2G 的 controller。broker 存储量按保留时长和吞吐规划。 + +### OpenSearch 规格 + +CPU 和内存是每个节点的 limit。 + +| 规格 | 节点 | 拓扑 | 每节点 CPU limit | 每节点内存 limit | 实测吞吐 | +| --- | --- | --- | --- | --- | --- | +| 小规格 | 3 | 3 个节点，承担全部角色 | 2C | 4G | 6,300 logs/s | +| 小规格 | 5 | 5 个节点，承担全部角色 | 2C | 4G | 9,900 logs/s | +| 大规格 | 3 + 5 | 3 个 master，5 个 data | master 2C / data 8C | master 4G / data 16G | 25,000 logs/s | +| 大规格 | 3 + 7 | 3 个 master，7 个 data | master 2C / data 8C | master 4G / data 16G | 30,000 logs/s | + +不要低于最小规格，当单个节点池无法承载数据量时，请改用大规格。如果实际存储低于 6,000 IOPS 和 250 MB/s 读写，请上调规格。 + +### 磁盘 + +请提供独占 SSD 存储，至少 6,000 IOPS 和 250 MB/s 读写，并按保留时长规划容量：大部分日志 7 天，Kubernetes 日志 30 天，事件和审计 180 天，计量 540 天。下面的示例按每个 ClickHouse Pod 200 Gi、每个 Kafka broker 200 Gi、每个 Kafka controller 20 Gi 规划；OpenSearch 的 master 与 data 节点池需要分别规划。 + +## 步骤 1：节点和磁盘 + +1. 选择用于运行存储集群的节点，不要与业务工作负载混部。 +2. 按[为日志存储规划基础设施节点](https://docs.alauda.cn/logging-service/4.3/how_to/infra_nodes.html)给这些节点打上基础设施节点标签，并添加对应的污点。下面的清单使用 `node-role.kubernetes.io/infra` 作为选择标签并容忍该污点；如果你的集群用别的 key，请在每个清单里同步改掉。 +3. 为每个节点挂载独占 SSD 作为持久化路径： + - 使用传统操作系统布局时，使用 `/cpaas/data/...`。 + - 在 Alauda OS 节点上只有 `/var/cpaas` 可写，因此使用 `/var/cpaas/data/...`。 +4. 确保该路径在节点重新纳管后仍然保留。 +5. OpenSearch 需要把 `vm.max_map_count` 设置为不小于 `262144`。OpenSearch Operator 会通过 init 容器设置它，因此这里通常不用处理；只有集群启用了受限的 Pod Security Admission 时，该 init 容器无法设置，才需要在每个节点上手工设置： + + ```bash + sudo sysctl -w vm.max_map_count=262144 + echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf + ``` +6. 创建存储 Pod 使用的目录并设置属主。下面的示例使用传统布局；在 Alauda OS 节点上请把 `/cpaas` 换成 `/var/cpaas`。 + + ```bash + # ClickHouse 以 uid 101 运行 + sudo mkdir -p /cpaas/data/clickhouse + sudo chown -R 101:101 /cpaas/data/clickhouse + + # Kafka 以 uid 1001 运行；每个 Pod 一个目录，见下文 + sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-3 + sudo chown -R 1001:1001 /cpaas/data/kafka + + # OpenSearch 以 uid 1000 运行 + sudo mkdir -p /cpaas/data/opensearch + sudo chown -R 1000:1000 /cpaas/data/opensearch + ``` + +7. 确定存储卷的供给方式： + +| 方式 | 适用场景 | 需要做什么 | +| --- | --- | --- | +| 静态本地卷 | 需要把每个 Pod 固定到指定节点，这也是基础设施节点方案通常采用的方式 | 创建一个不带 provisioner 的 StorageClass，并按预期的 Pod 数量预先创建 PV，每个 PV 通过 `nodeAffinity` 和 `local.path` 指向上面的目录 | +| 动态供给 | 平台已提供块存储 provisioner | 创建 StorageClass，由 PVC 动态绑定；确认 provisioner 支持 `ReadWriteOnce` 块卷并满足上述吞吐要求 | + +使用静态本地卷时，请创建一个 StorageClass 并按每个 Pod 一个 PV 预先创建，同时**把每个 PV 预留给它自己的 PVC**。`local` 卷不能跟随 Pod 迁移：如果不做预留，某个 PVC 可能绑走为其他 Pod 或其他组件准备的卷；实例删除重建后，各 Pod 之间也可能互相绑错盘。用 `spec.claimRef` 预留之后，这个卷只会匹配它上面写明的那个 PVC。 + +请为每个组件使用独立的 StorageClass。下面的示例创建 ClickHouse 那个，并为第一个 ClickHouse 副本预留一个卷。PV 数量等于你规划的 Pod 总数：ClickHouse 为 `shardsCount × replicasCount`，Kafka 为 `replicas + controller.replicas`，OpenSearch 为各节点池 `replicas` 之和。 + +ClickHouse 和 OpenSearch 的 PVC 名称是确定的，因此可以在实例创建之前就把卷预留好： + +| 组件 | 副本 `` 的 PVC 名称 | +| --- | --- | +| ClickHouse | `data-volumeclaim-template-chi-----0` | +| OpenSearch | `data---` | +| Kafka | `data--broker--`，需要先创建实例再读取名称，见步骤 4 | + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local-clickhouse + labels: + # 在项目命名空间中是必需的：没有这个授权， + # pvc-validator 准入 webhook 会拒绝所有使用该 StorageClass 的 PVC。 + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-clickhouse-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-clickhouse + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + local: + path: /cpaas/data/clickhouse + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +请把 YAML 保存为 `local-storage.yaml` 并应用： + +```bash +kubectl apply -f local-storage.yaml +``` + +请为每个 Pod 重复 `PersistentVolume` 部分，`metadata.name` 和 `local.path` 各不相同，`values` 填该 Pod 所在节点的 IP。同一节点上跑多个同类 Pod 时，每个 Pod 用独立目录，例如 `/cpaas/data/clickhouse-0` 和 `/cpaas/data/clickhouse-1`，属主按步骤 6 设置。 + +`local` 卷上的 `capacity.storage` 只是匹配用的元数据，不是配额：Pod 完全可以写满底层磁盘。请把它设成真实可用容量，并在存储侧同时配置保留策略。 + +## 步骤 2：安装 Operator + +请从平台市场安装这三个 Operator。下面创建的存储和消息资源都位于 `cpaas-system`，因此每个 Operator 都必须能协调该命名空间中的资源。 + +| Operator | Subscription 所在命名空间 | Operator 必须 watch 的命名空间 | +| --- | --- | --- | +| `clickhouse-operator` | `cpaas-system` | `cpaas-system` | +| Alauda Kafka Operator（`strimzi-kafka-operator`） | `kafka-system` | 所有命名空间 | +| `opensearch-operator`（仅目标存储为 OpenSearch 时） | `opensearch-operator` | 所有命名空间 | + +- 只安装缺失的 Operator。如果集群上已经装了某一个（例如由更早的版本装入），请沿用已有的，不要再装第二份：同一个 Operator 的两份实例会同时写 `cpaas-system` 下的同一批资源。此时改为检查它的 watch 范围，必要时放开。 +- 不要在 `cpaas-system` 中创建 OperatorGroup。平台已在其中创建了一个，再创建一个会导致平台拒绝该命名空间下的所有 Subscription，包括平台自己的。 +- `kafka-system` 和 `opensearch-operator` 的 OperatorGroup 必须不包含 `spec.targetNamespaces`。如果已经存在一个只作用于自身命名空间的 OperatorGroup，请删除该字段，并等待 Operator Pod 重启。 + + ```bash + kubectl -n kafka-system patch operatorgroup kafka-system \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + ``` + +继续之前请校验每个 Operator： + +```bash +# ClickHouse +kubectl get crd clickhouseinstallations.clickhouse.altinity.com +kubectl -n cpaas-system get deploy clickhouse-operator + +# Kafka +kubectl get crd rdskafkas.middleware.alauda.io +kubectl -n kafka-system get deploy strimzi-cluster-operator + +# OpenSearch（仅当目标存储为 OpenSearch 时） +kubectl get crd opensearchclusters.opensearch.opster.io +kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager +``` + +:::warning +Operator 如果未 watch `cpaas-system`，会静默忽略下面的资源：没有 status、没有事件、也没有 Pod。继续之前请确认 Deployment 已就绪，且 Kafka 和 OpenSearch 的 OperatorGroup 覆盖所有命名空间。 +::: + +## 步骤 3：创建 ClickHouse 集群 + +目标存储为 OpenSearch 时请跳过本步骤。 + +### 3.1 创建密码 Secret + +实例定义了两个账号：用于管理的 `admin`，以及供日志组件使用的 `platform-logging`。两者的密码都来自 Secret，因此请在创建实例之前先把两个 Secret 建好。 + +```bash +kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ + --from-literal=password="$(openssl rand -hex 16)" +kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 3.2 创建 Keeper 客户端 Service（三节点及以上） + +所有规格都需要 Keeper，单节点也一样，因为日志组件创建的是 `ReplicatedMergeTree` 表。单节点规格把 Keeper 运行在自己的 ClickHouse Pod 内，靠 3.3 中的 `keeper_server/*` 配置实现，因此单节点请跳过本步骤。 + +三节点及以上时，由 ClickHouse Pod 自身组成 Keeper 仲裁集群：每个 ClickHouse Pod 同时也是一个 Keeper 成员。ClickHouse 通过一个 headless Service 访问该仲裁集群，该 Service 会选中本次安装中所有就绪的 Pod。 + +请把 YAML 保存为 `cpaas-clickhouse-keeper-service.yaml` 并应用： + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + clusterIP: None + type: ClusterIP + ports: + - name: keeper + port: 9181 + protocol: TCP + targetPort: 9181 + selector: + clickhouse.altinity.com/chi: cpaas-clickhouse + clickhouse.altinity.com/namespace: cpaas-system + clickhouse.altinity.com/ready: "yes" + clickhouse.altinity.com/role: keeper +``` + +```bash +kubectl apply -f cpaas-clickhouse-keeper-service.yaml +``` + +其中 `chi`、`namespace`、`ready` 三个标签由 Operator 打上，`role: keeper` 标签来自 3.3 的 pod template。 + +### 3.3 创建 ClickHouseInstallation + +cluster 名称固定为 `replicated`，需与日志组件保持一致；`shardsCount` 和 `replicasCount` 按步骤 0 的规格设置。下面两份清单请按你选的规格二选一执行。 + +请把 `` 替换为平台 middleware 包发布的 ClickHouse server 镜像，例如 `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`。 + +**单节点。** Keeper 通过 `keeper_server/*` 配置运行在 ClickHouse Pod 内。 + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # 管理员密码来自上面创建的 Secret + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # 日志组件使用的账号。与 admin 一样在这里声明， + # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + clusters: + - name: replicated # 连接 Secret 中需要复用该名称 + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # 取自规格：1、1、2 或 3 + replicasCount: 1 # 示例为单 Pod；三节点及以上请改为 3 + + settings: + default_database: observability # 连接 Secret 中需要复用该名称 + merge_tree/materialize_ttl_recalculate_only: "1" + # 自身可观测性系统表会无限增长，最终写满数据盘。 + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + # 单节点规格使用同 Pod 内嵌 Keeper。三节点及以上请改用下面那份清单， + # 它会为每个 ClickHouse Pod 都运行一个 Keeper。 + keeper_server/tcp_port: "9181" + keeper_server/server_id: "1" + keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log + keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots + keeper_server/coordination_settings/operation_timeout_ms: "10000" + keeper_server/coordination_settings/session_timeout_ms: "30000" + keeper_server/raft_configuration/server/id: "1" + keeper_server/raft_configuration/server/hostname: localhost + keeper_server/raft_configuration/server/port: "9234" + + zookeeper: + nodes: + - host: localhost + port: 9181 + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: keeper + containerPort: 9181 + - name: raft + containerPort: 9234 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # 来自步骤 1 +``` + +请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +**三节点及以上。** 请改用下面这份清单，不要用上面那份。Keeper 运行在每一个 ClickHouse Pod 内，Pod 之间自行组成仲裁集群，因此整个实例仍然只是一个 `ClickHouseInstallation`。 + +静态 Keeper 配置通过 cluster 的 `files` 注入，并用 `include_from` 引入生成出来的文件；与身份相关的部分（`server_id` 和成员列表）由 init 容器按 Pod 生成到内存 `emptyDir` 中。init 容器里的 `SHARDS_COUNT`、`REPLICAS_COUNT` 必须与 `layout.shardsCount`、`layout.replicasCount` 保持一致，否则成员列表不完整，仲裁永远组不起来。 + +readiness 探针探测的是 Raft 端口，这是必需的：默认的 HTTP 探针要等 ClickHouse 开始提供服务才会成功，而 ClickHouse 又必须等 Keeper 仲裁集群就绪才能完成启动，于是 Operator 会一直等第一个副本，永远不创建其余副本。 + +Keeper 的 `path` 位于 `/var/lib/clickhouse` 之下，也就是挂载的数据卷内，因此 Keeper 的日志与快照和 ClickHouse 数据一样保存在持久卷上。不要把它移到该挂载点之外：放在容器文件系统里的 Keeper 状态会在 Pod 每次重启时丢失。 + +`wait-for-self-dns` init 容器会等待 Pod 能解析自己的 headless Service 名称。没有它时，如果某个 Pod 在自己的 DNS 记录发布之前启动，它的分布式 DDL worker 会基于一个解析不了的主机名完成初始化并且不再重试：`CREATE TABLE ... ON CLUSTER` 在其他副本上执行成功，而该副本会静默漏掉这条语句。 + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # 管理员密码来自上面创建的 Secret + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # 日志组件使用的账号。与 admin 一样在这里声明， + # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper # 3.2 中创建的 Service + port: 9181 + + settings: + default_database: observability # 连接 Secret 中需要复用该名称 + merge_tree/materialize_ttl_recalculate_only: "1" + # 自身可观测性系统表会无限增长，最终写满数据盘。 + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + + clusters: + - name: replicated # 连接 Secret 中需要复用该名称 + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # 取自规格：1、2 或 3 + replicasCount: 3 # 三节点及以上为 3 + shards: + - files: + keeper_config.xml: | + + /tmp/clickhouse/keeper_dynamic_configuration.xml + + /var/lib/clickhouse/coordination + 9181 + * + + information + + + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + metadata: + labels: + clickhouse.altinity.com/role: keeper # 由 3.2 的 Service 选中 + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + env: + - name: RAFT_PORT + value: "9444" + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: ch-keeper + containerPort: 9181 + - name: raft + containerPort: 9444 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + - name: keeper-dynamic-config + mountPath: /tmp/clickhouse + readinessProbe: + tcpSocket: + port: 9444 + initialDelaySeconds: 10 + timeoutSeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + initContainers: + - name: wait-for-self-dns + image: + command: + - /bin/bash + - -c + - | + SELF="$(hostname -s).$(hostname -d)" + MY_IP="$(hostname -i)" + for i in $(seq 1 60); do + RESOLVED=$(getent hosts "$SELF" 2>/dev/null | awk '{print $1}' | head -1) + if [ "$RESOLVED" = "$MY_IP" ]; then + echo "resolved $SELF to $MY_IP after ${i}s" + exit 0 + fi + sleep 2 + done + echo "still unresolved after 120s, continuing" + exit 0 + - name: keeper-config-initializer + image: + env: + - name: RAFT_PORT + value: "9444" + - name: SHARDS_COUNT + value: "1" # 与 layout.shardsCount 保持一致 + - name: REPLICAS_COUNT + value: "3" # 与 layout.replicasCount 保持一致 + command: + - /bin/bash + - -c + - | + set -euo pipefail + OUT="/tmp/config/keeper_dynamic_configuration.xml" + HOST=$(hostname -s) + DOMAIN=$(hostname -d) + if [[ $HOST =~ (.*)-([0-9]+)-([0-9]+)-([0-9]+)$ ]]; then + SHARD=${BASH_REMATCH[2]} + REPLICA=${BASH_REMATCH[3]} + else + echo "Failed to parse shard/replica from hostname $HOST"; exit 1 + fi + if [[ $DOMAIN =~ ^(.*)-([0-9]+)-([0-9]+)\.(.*)$ ]]; then + DOMAIN_NAME=${BASH_REMATCH[1]} + DOMAIN_SUFFIX=.${BASH_REMATCH[4]} + else + echo "Failed to parse domain $DOMAIN"; exit 1 + fi + MY_ID=$((SHARD * REPLICAS_COUNT + REPLICA + 1)) + KEEPER_ID=1 + { + echo "" + echo " " + echo " ${MY_ID}" + echo " " + for (( i=0; i" + echo " ${KEEPER_ID}" + echo " ${DOMAIN_NAME}-${i}-${j}${DOMAIN_SUFFIX}" + echo " ${RAFT_PORT}" + echo " " + KEEPER_ID=$((KEEPER_ID + 1)) + done + done + echo " " + echo " " + echo "" + } > "$OUT" + echo "Keeper dynamic configuration generated for server_id=${MY_ID}" + volumeMounts: + - name: keeper-dynamic-config + mountPath: /tmp/config + volumes: + - name: keeper-dynamic-config + emptyDir: + medium: Memory + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # 来自步骤 1 +``` + +请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +`default_database: observability` 会让 ClickHouse 在启动时创建 `observability` 库，这里不需要再建库。 + +### 3.4 等待集群就绪 + +```bash +kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ + -o jsonpath='{.status.status}{"\n"}' # 反复执行，直到变为：Completed + +kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse +``` + +所有 ClickHouse Pod 必须处于 `Running` 且就绪，且每个 PVC 都必须是 `Bound`。StorageClass 不存在或无法绑定时不会有任何 Pod，也不会有报错，因此不能只看 `ClickHouseInstallation` 的 status。 + +三节点及以上请先确认 Keeper 仲裁集群再继续。Keeper 运行在 ClickHouse Pod 内，因此逐个副本检查一个 Pod： + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +``` + +三条命令的结果必须是 1 个 `leader`、2 个 `follower`。同时确认 ClickHouse 是通过 3.2 的 Service 访问仲裁集群的： + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# 期望：cpaas-clickhouse-keeper 9181 +``` + +### 3.5 读取日志账号密码 + +该账号已在 3.3 应用的 `ClickHouseInstallation` 中声明，因此这里不需要创建账号，也没有 `GRANT` 语句要执行：它和 `admin` 一样是服务端配置用户，自带日志组件所需的权限。 + +请读取它的密码，供步骤 6 记录连接信息。3.6 会复用 `CH_POD` 与 `LOG_PASSWORD` 变量，因此两步请在同一个 shell 中执行。 + +```bash +CH_POD="$(kubectl -n cpaas-system get pod \ + -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o jsonpath='{.items[0].metadata.name}')" +LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "platform-logging password: $LOG_PASSWORD" +``` + +### 3.6 校验 + +请在同一个 shell 中执行（3.5 定义的 `$CH_POD` 和 `$LOG_PASSWORD` 还在）： + +```bash +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" + +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "DROP TABLE observability.__perm_check" +``` + +两条命令都必须成功，建表失败说明该账号无法管理表结构。 + +记录以下信息： + +| 值 | 从哪里获取 | +| --- | --- | +| 连接地址 | 暴露 `8123` 的集群 Service，可用 `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse` 查看 | +| Cluster 名称 | `spec.configuration.clusters[].name` | +| 数据库 | `spec.configuration.settings.default_database` | +| 分片数 / 副本数 | `shardsCount` / `replicasCount` | +| 用户名 / 密码 | 上面创建的账号 | + +## 步骤 4：创建 Kafka 服务 + +### 4.1 创建 SASL 密码 Secret + +在 Alauda OS 节点或其他启用了 FIPS 的主机上，密码长度必须不少于 32 个字符。 + +```bash +kubectl -n cpaas-system create secret generic platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 4.2 创建 broker 集群 + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + mode: KRaft + version: 4.2.0 # Alauda Kafka Operator 支持的最低版本 + replicas: 3 + resources: + limits: { cpu: "2", memory: 4Gi } # 取自规格 + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: + deleteClaim: false + controller: + replicas: 3 + roles: ["controller"] # 必需：缺少时 node pool 会被拒绝 + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.hours: "48" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # 必需：由它创建 4.5 中的 topic + userOperator: {} # 必需：由它创建 4.4 中的 SASL 用户 +``` + +请把 YAML 保存为 `cpaas-kafka.yaml` 并应用： + +```bash +kubectl apply -f cpaas-kafka.yaml +``` + +实例会立即创建自己的 PVC。在为其预留卷之前，这些 PVC 会一直处于 `Pending`：broker 的 PVC 名称里含有按实例生成的 hash，所以卷无法提前准备： + +```bash +kubectl -n cpaas-system get pvc \ + -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' +``` + +把这六个名称原样抄下来（三个 `...-broker-...`、三个 `...-controller-...`），并为每个 PVC 创建一个预绑定的卷。`capacity.storage` 填该 PVC 申请的容量，`local.path` 指向该 Pod 的目录，并用节点亲和把它钉在对应节点上： + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-kafka-broker-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-kafka + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-cpaas-kafka-broker--0 + local: + path: /cpaas/data/kafka/broker-0 + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +请把 YAML 保存为 `cpaas-kafka-volumes.yaml`，为全部六个 PVC 重复 `PersistentVolume` 部分后应用。卷一旦存在，PVC 立即绑定，broker 与 controller 随之启动： + +```bash +kubectl apply -f cpaas-kafka-volumes.yaml +``` + +以下配置不能省略： + +| 配置项 | 为什么必须设置 | +| --- | --- | +| `message.max.bytes: "10485760"` | 审计数据的批大小约为 1.1–1.5 MiB，Kafka 默认的 1 MiB 会拒绝所有审计批次。 | +| `replica.fetch.max.bytes: "10485760"` | 必须不小于 `message.max.bytes`，否则副本同步会停滞。 | +| `auto.create.topics.enable: "false"` | 避免拼错的 topic 名称被自动创建并静默接收数据。 | +| `entityOperator.topicOperator` / `userOperator` | 缺少它们时，下一步中的 `RdsTopic` 和 `RdsKafkaUser` 不会生效到 broker。 | + +### 4.3 等待 broker 集群就绪 + +```bash +kubectl -n cpaas-system get rdsKafka cpaas-kafka \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' +kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka +``` + +等待 `Ready` 条件变为 `True`，且所有 broker Pod 处于 `Running`。Kafka Operator 会自动施加硬反亲和，因此三个 broker 必须落在三个不同节点上： + +```bash +kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +``` + +必须出现三个不同的节点名：硬反亲和要求集群至少有 3 个可调度节点。 + +确认必需配置已下发到 broker： + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ + grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes" /tmp/strimzi.properties +``` + +两个值都必须是 `10485760`，否则不要继续。这些配置以静态配置形式写入 broker，`kafka-configs.sh --describe` 看不到它们。 + +### 4.4 创建 SASL 用户及其 ACL + +日志组件使用同一个账号，需要访问三个 topic、消费组以及 broker 元数据。 + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser +metadata: + name: platform-logging + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + # 三个 topic + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } + # 日志链路使用的消费组 + - host: "*" + operation: All + resource: { type: group, name: alauda_log, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_event, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_audit, patternType: literal } + # LogForward 使用的消费组前缀 + - host: "*" + operation: All + resource: { type: group, name: "logforward-", patternType: prefix } + # 日志查询服务使用的消费组前缀 + - host: "*" + operation: All + resource: { type: group, name: "razor-", patternType: prefix } + # broker 元数据 + - host: "*" + operation: All + resource: { type: cluster, name: kafka-cluster, patternType: literal } +``` + +请把 YAML 保存为 `platform-logging-user.yaml` 并应用： + +```bash +kubectl apply -f platform-logging-user.yaml +``` + +这九条都必须配置，`operation: All` 已覆盖这些条目所需的读和 describe 权限。请校验： + +```bash +kubectl -n cpaas-system get rdskafkauser platform-logging \ + -o jsonpath='{.status.phase}{"\n"}' # 期望：Active +kubectl -n cpaas-system get secret platform-logging +``` + +消费组名称使用下划线（`alauda_log`），而 topic 名称使用大写字母和下划线（`ALAUDA_LOG_TOPIC`）。 + +### 4.5 创建三个 topic + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC # broker 侧名称，必须与 ACL 和连接 Secret 一致 + partitions: 30 # 消费并发度的上限 + replicas: 3 + config: + retention.ms: "172800000" # 48 小时 + segment.bytes: "1073741824" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "1073741824" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "1073741824" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +``` + +请把 YAML 保存为 `alauda-topics.yaml` 并应用： + +```bash +kubectl apply -f alauda-topics.yaml +``` + +资源名必须是合法的 DNS 名称。`spec.topicName` 是 broker 侧名称，必须与上面的 ACL 条目一致。 + +示例中的保留时间为 48 小时。只按时间保留并不能限制磁盘占用：一波流量峰值可能在保留窗口到期前就把 broker 卷写满。请按「保留窗口内的峰值速率」规划 broker 卷容量，或者给三个 topic 都加上 `retention.bytes`。`retention.bytes` 是按分区生效的，因此 broker 卷需要能容纳 `分区数 × retention.bytes`。 + +### 4.6 端到端校验 Kafka 服务 + +先在 broker Pod 内根据用户 Secret 生成客户端配置。broker 监听端口要求 SASL 认证，下面的命令都需要它。 + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ + sh -c 'cat > /tmp/logging-client.properties' <-kafka-bootstrap.cpaas-system.svc:9093`，不使用 TLS 的 SASL 为 `:9092` | +| Cluster 名称 | `RdsKafka` 资源的 `metadata.name` | +| 用户名 / 密码 | `RdsKafkaUser` 的名称及其密码 | +| Topic | `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC`、`ALAUDA_AUDIT_TOPIC` | +| CA 证书 | 仅在用 `9093` 的 TLS 监听端口时需要。Kafka Operator 会把它放在 `cpaas-system` 下名称以 `-cluster-ca-cert` 结尾的 Secret 中 | + +## 步骤 5：创建 OpenSearch 集群 + +目标存储为 ClickHouse 时请跳过本步骤。 + +### 5.1 创建集群 + +按步骤 0 的规格设置节点池。示例为 3 + 5：三个 master、五个 data。小规格只用一个节点池，`roles` 设为 `[cluster_manager, data]`，`replicas` 设为 3 或 5。 + +```yaml +apiVersion: opensearch.opster.io/v1 +kind: OpenSearchCluster +metadata: + name: cpaas-opensearch + namespace: cpaas-system +spec: + general: + serviceName: cpaas-opensearch + httpPort: 9200 + version: 3.7.0 + security: + tls: + http: + generate: true + transport: + generate: true + perNode: true + nodePools: + - component: masters + replicas: 3 + diskSize: 100Gi + roles: + - cluster_manager + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # 来自步骤 1 + resources: + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + requests: + cpu: "1" + memory: 2Gi + - component: data + replicas: 5 + diskSize: 800Gi + roles: + - data + - ingest + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # 来自步骤 1 + resources: + limits: + cpu: "8" # 取自规格 + memory: 16Gi # 取自规格 + requests: + cpu: "2" + memory: 8Gi + dashboards: + replicas: 0 +``` + +请把 YAML 保存为 `cpaas-opensearch.yaml` 并应用： + +```bash +kubectl apply -f cpaas-opensearch.yaml +``` + +等待集群健康： + +```bash +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' # 期望：green +``` + +节点启动和分片初始化的过程中，健康状态会先显示 `unknown`，随后是 `yellow`。请等到 `green`。 + +后文的这些调用都使用管理员账号 `admin` 认证。该账号由 Operator 创建，凭据保存在 `-admin-password` Secret 中，请从这里读取密码，填到后文所有 `` 的位置。默认账号密码是 `admin` / `admin`，也就是示例假定的取值： + +```bash +OS_ADMIN_USER="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.username}' | base64 -d)" +OS_ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "$OS_ADMIN_USER / $OS_ADMIN_PASSWORD" +``` + +生产环境请先修改该密码，并且要和 Operator 侧保持一致，否则 Operator 自身的健康检查会失败；平台 OpenSearch 知识库里有完整改密流程（把新密码的 hash 写入 `internal_users.yml` 并生成 `securityconfig-secret`，同时创建匹配的 `admin-credentials-secret`，再更新集群）。日志账号的 `` 请填不少于 32 个字符的值。 + + +### 5.2 可选：安装中文分词插件 + +`analysis-ik` 是可选组件。不安装时使用 standard 分析器，中文不分词；安装后中文会分词。 + +请在 `OpenSearchCluster` 的 spec 中配置该插件。节点每次启动时，operator 会把 `pluginsList` 中的每一项交给 `opensearch-plugin install` 执行，因此插件在 Pod 重启、节点替换后依然存在。手工进容器安装不算数：插件写在数据卷之外，Pod 一重启就没了。 + +| 字段 | 作用 | +| --- | --- | +| `spec.general.pluginsList` | 在所有 OpenSearch 节点上安装插件 | +| `spec.bootstrap.pluginsList` | 在负责组建集群的 bootstrap Pod 上安装插件。只要是在新建集群时就要启用插件，这个字段必须一起配置，否则集群初始化可能失败。 | + +如果希望在创建集群时就带上插件，请在 apply 5.1 的 `cpaas-opensearch.yaml` 之前把这两个字段加进去： + +```yaml +spec: + general: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" + bootstrap: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" +``` + +如果集群已经建好且没装插件，改用下面的 patch 命令即可，operator 会滚动重启节点完成安装： + +```bash +kubectl -n cpaas-system patch opensearchcluster cpaas-opensearch --type=merge -p '{"spec":{"general":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]},"bootstrap":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]}}}' +``` + +插件版本必须与 `spec.general.version` 一致。URL 返回 404 会导致所有节点无法启动。集群无法访问外网时，请先把 zip 放到内网服务器，并把 URL 换成内网地址。 + +校验每个节点都已安装： + +```bash +for p in $(kubectl -n cpaas-system get pod -l opster.io/opensearch-cluster=cpaas-opensearch -o jsonpath='{.items[*].metadata.name}'); do echo -n "$p: "; kubectl -n cpaas-system exec $p -c opensearch -- bin/opensearch-plugin list | grep -c '^analysis-ik'; done +``` + +每个节点都必须输出 `1`。随后确认分析器能对中文分词。请把 `` 替换为集群 Service 地址，例如 `https://cpaas-opensearch.cpaas-system.svc:9200`： + +```bash +curl -sk -u "admin:" -X POST "https:///_analyze" \ + -H 'Content-Type: application/json' \ + -d '{"analyzer":"ik_smart","text":"自然语言处理技术"}' +``` + +返回结果必须包含多个词元，例如 `自然语言`、`处理`、`技术`。 + +### 5.3 创建日志组件使用的账号 + +该账号需要读写日志索引、创建日志组件使用的索引模板，并管理其生命周期策略。security 插件已启用，因此需要通过其 REST API 创建角色、用户和角色映射。这些调用同样使用管理员账号 `admin`，密码就是 5.1 中读到的那个。 + +```bash +OS="https://" # 集群 Service，例如 https://cpaas-opensearch.cpaas-system.svc:9200 + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/roles/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{ + "cluster_permissions": [ + "cluster:monitor/*", + "cluster:admin/opendistro/ism/policy/*", + "indices:admin/index_template/put", + "indices:admin/index_template/get", + "indices:admin/template/put", + "indices:admin/template/get" + ], + "index_permissions": [{ + "index_patterns": ["log-*", "event-*", "audit-*", "meter-*"], + "allowed_actions": [ + "indices:admin/create", + "indices:admin/mapping/put", + "indices:data/write/*", + "indices:data/read/*" + ] + }] + }' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/internalusers/platform-logging" \ + -H 'Content-Type: application/json' \ + -d '{"password":"","backend_roles":[]}' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/rolesmapping/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{"users":["platform-logging"]}' +``` + +模板和策略权限属于集群级权限。把 `indices:admin/index_template/put` 和 `indices:admin/template/put` 写在 `index_permissions` 下不会生效，日志组件随后会因无权限而无法创建索引模板。缺少角色映射时，账号可以认证但没有任何权限。 + +用新账号校验： + +```bash +curl -sk -u "platform-logging:" -X PUT "$OS/_index_template/perm-check" \ + -H 'Content-Type: application/json' \ + -d '{"index_patterns":["log-perm-check-*"],"template":{"settings":{"number_of_shards":1}}}' + +curl -sk -u "platform-logging:" -X POST "$OS/log-perm-check/_doc" \ + -H 'Content-Type: application/json' -d '{"check":1}' + +curl -sk -u "platform-logging:" "$OS/_index_template/perm-check" +``` + +三条命令都必须成功。出现 `security_exception` 且提示 `no permissions for [...]`，说明角色或角色映射不完整。最后用管理员账号清理： + +```bash +curl -sk -u "admin:" -X DELETE "$OS/_index_template/perm-check" +curl -sk -u "admin:" -X DELETE "$OS/log-perm-check" +``` + +记录以下信息： + +| 值 | 从哪里获取 | +| --- | --- | +| 连接地址 | Service 地址或负载均衡地址，例如 `https://:9200` | +| 用户名 / 密码 | 上面创建的账号 | +| CA 证书 | 仅在连接地址使用私有 CA 时需要。Operator 会把它创建在 `cpaas-system` 下的 Secret `cpaas-opensearch-ca` 中 | + +平台下发的索引模板为 1 分片 1 副本。如果你的高可用策略需要不同的取值，请下发一个优先级更高的可组合模板，并用 `GET /_index_template` 确认结果。已创建的索引会保持创建时的设置。 + +## 步骤 6：记录连接信息 + +请记录下表各项。 + +| 值 | ClickHouse | OpenSearch | Kafka | +| --- | --- | --- | --- | +| 连接地址 | 必需 | 必需 | 必需 | +| Cluster 名称 | 必需 | — | 必需 | +| 数据库 | 必需 | — | — | +| 分片数 / 副本数 | 必需 | — | — | +| 用户名 | 必需 | 必需 | 必需 | +| 密码 | 必需 | 必需 | 必需 | +| Topic | — | — | 必需 | +| CA 证书 | 连接地址使用私有 CA 时必需 | 连接地址使用私有 CA 时必需（用于历史数据迁移） | 连接地址使用私有 CA 时必需 | + +## 环境检查清单 + +| 检查项 | 期望结果 | +| --- | --- | +| 节点和磁盘 | 已使用独占节点并打好标签与污点，已挂载 SSD，目录已按正确的属主创建，且清单中已容忍该污点 | +| StorageClass | 已存在，并且能够为存储集群绑定存储卷 | +| Operator | ClickHouse、Kafka 以及（目标为 OpenSearch 时）OpenSearch 的 Operator 已就绪，其 CRD 已存在，且都能 watch `cpaas-system` | +| ClickHouse | `status.status` 为 `Completed`，所有 Pod 就绪，日志账号可以建表和删表；三节点及以上时 Keeper 仲裁集群为 1 个 leader、2 个 follower | +| OpenSearch | 集群健康状态为 `green`，日志账号可以管理索引模板；启用插件时，每个节点都能看到 `analysis-ik` 且能对中文分词 | +| Kafka broker | 集群就绪，`message.max.bytes` 和 `replica.fetch.max.bytes` 均为 `10485760` | +| Kafka 用户和 ACL | `RdsKafkaUser` 为 `Active`，九条 ACL 均已配置 | +| Kafka topic | broker 上已存在三个 topic，且分区数和保留时间符合预期 | +| Kafka 连通性 | 已使用日志账号成功生产和消费一条记录 | +| 信息记录 | 连接地址、cluster、数据库、拓扑、凭据、topic 和 CA 证书均已记录 | + +任何一项不通过，都必须在日志组件连接该存储之前修复。 From 14bf53158a13423e44739066c465c97f08037d8a Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 13:27:01 +0800 Subject: [PATCH 34/38] docs: split the migration out of the upgrade chapter The upgrade chapter was carrying the whole historical data migration: the LegacyESMigration manifests for both targets, the capture and copy sequence, the batch and concurrency options, and the verification. That made the chapter read as two procedures in one. Move it to its own chapter, Historical Data Migration. The upgrade chapter keeps the upgrade itself: install the new data path with the annotation, create the migration resource when history is required, uninstall the legacy plugin, and keep the protected volumes. Where migration applies it links to the new chapter instead of repeating it. The new chapter also states the supported scope, which the upgrade chapter never did: - the source is the legacy Elasticsearch plugin's PVCs, discovered automatically while the legacy StatefulSet is present or listed explicitly afterwards; - the data is the indices selected by indexScope, which must match at least one index; - the target is ClickHouse or OpenSearch, using the same connection Secret as the PlatformLogForward; - the boundary is captured when the resource is created and the copy runs after the uninstall; - ClickHouse runs one job at a time, OpenSearch accepts two; - and the migrated time range has to fit inside the target retention, because the migration does not widen or bypass the TTL that PlatformLogForward and razor already applied. --- docs/en/migration/index.mdx | 127 ++++++++++++++++++++++++++++++++++ docs/en/upgrade/index.mdx | 134 ++---------------------------------- 2 files changed, 134 insertions(+), 127 deletions(-) create mode 100644 docs/en/migration/index.mdx diff --git a/docs/en/migration/index.mdx b/docs/en/migration/index.mdx new file mode 100644 index 0000000..0e4dcd2 --- /dev/null +++ b/docs/en/migration/index.mdx @@ -0,0 +1,127 @@ +--- +weight: 16 +--- + +# Historical Data Migration + +This chapter migrates the historical log, event, and audit data that the legacy Elasticsearch storage plugin holds into the ClickHouse or OpenSearch target that the new data path uses. Run it as part of the upgrade; the upgrade chapter links to it at the point where it applies. + +## Supported scope + +| Item | Supported | +| --- | --- | +| Source | The PVCs of the legacy Elasticsearch storage plugin. Leave `source.pvcRefs` out and the operator discovers them while the legacy StatefulSet and its PVCs are still present; list them explicitly when discovery is no longer possible, for example after the plugin is uninstalled. | +| Data | The indices selected by `source.indexScope`. It accepts exact index names and globs and must match at least one index; an empty list is rejected. | +| Target | `clickhouse` or `opensearch`. `target.secretRef` uses the same connection Secret as the `PlatformLogForward`. | +| Timing | The migration boundary is captured when you create the resource, and the copy runs after the legacy plugin is uninstalled. | +| Concurrency | ClickHouse targets always run one job at a time. OpenSearch targets accept `options.maxConcurrentJobs: 2`; the default is `1`. | +| Batch size | `options.batchSize` accepts `1..100000`; `options.syncIntervalSeconds` sets the minimum wait between batches. | + +## Before you start + +1. The new data path exists and is `Ready`; its `PlatformLogForward` points at the target you migrate into. Follow the upgrade chapter through the step that installs the new data path first. +2. The migration worker image for this release is available, with its complete registry, tag, or digest. `spec.image` is required; the operator keeps using the image the resource was created with unless you update it. +3. The target account can create and write the tables or indices the migration needs, and it is the same account the connection Secret carries. +4. **The migrated time range is still inside the retention of the target tables.** The migration does not widen, pause, or bypass the retention that the `PlatformLogForward` and razor have already applied, so records older than the target TTL can be written and then removed by the normal merge. Confirm the range first, or raise the target retention and confirm the table definitions took effect before you create this resource. + +## Step 1: Create the migration resource + +Create the `LegacyESMigration` **before** you uninstall the legacy plugin. Creating it earlier is what records the final source state and the source volumes; a resource created after the uninstall has to name the source volumes explicitly and cannot use that final capture. + +### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required; migration image provided for this release, with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # Must match the PlatformLogForward target + secretRef: + name: platform-default-os-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000; default 250 + syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (no fixed wait for OpenSearch) + maxConcurrentJobs: 1 # Concurrent jobs; default 1, up to 2 for OpenSearch +``` + +### Target ClickHouse + +Change `target.type` to `clickhouse` and point the Secret at `platform-default-ch-conn`. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required; migration image provided for this release, with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # Must match the PlatformLogForward target + secretRef: + name: platform-default-ch-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000; default 250 + syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (1 second for ClickHouse) + maxConcurrentJobs: 1 # Concurrent jobs; ClickHouse is always serial (1) +``` + +Save the YAML as `legacy-es-migration.yaml` and apply it: + +```bash +kubectl apply -f legacy-es-migration.yaml +``` + +## Step 2: Wait for the capture to finish + +The copy does not start until the legacy plugin is uninstalled, so the phase stops at `PrecaptureReady` for now: + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history -w +``` + +In this flow, data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 3 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. + +The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. + +Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. + +## Step 3: Verify the migration + +Once the legacy plugin is uninstalled, the worker copies the data. Watch the same resource until it succeeds: + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}' # wait for: Succeeded +``` + +If the phase is `Blocked`, do not delete or recreate the resource; read the condition message and contact support. + +Then confirm the migrated data in the target: run the same log, event, and audit queries you use in production and check that the historical range is present. + +## Retained volumes + +Keep the protected source volumes until the verification passes and the approved change window closes. `status.phase: Succeeded` is not by itself approval to release them. + diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index fa9e3dd..1fbe187 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -100,9 +100,9 @@ Use this order. Each gate must pass before the next step starts. | Step | Where | Action | Gate to continue | | --- | --- | --- | --- | | 1 | Workload cluster | Install the new data path: connection Secrets plus `PlatformLogForward` with the upgrade annotation | `Phase=Ready` and `LegacyESUpgradeCompleted` | -| 2 | Workload cluster | If history is needed, create `LegacyESMigration` | `PrecaptureReady` (or `Succeeded` on an existing run) | +| 2 | Workload cluster | If history is needed, create `LegacyESMigration` as described in [Historical Data Migration](../migration/index.mdx) | `PrecaptureReady` (or `Succeeded` on an existing run) | | 3 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | -| 4 | Workload cluster | Observe the same `LegacyESMigration` | `Phase=Succeeded` and target queries pass | +| 4 | Workload cluster | Observe the same `LegacyESMigration` until it succeeds | `Phase=Succeeded` and target queries pass | | 5 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | ## Upgrade Procedure @@ -147,107 +147,13 @@ Once this step is complete, produce or locate new log, event, and audit records, If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform drains the queued data through the legacy path automatically and records `LegacyKafkaDrained` when the old consumer lag reaches zero; `LegacyESUpgradeCompleted` is the gate for this procedure. -### Step 2: Prepare the historical data migration (optional) +### Step 2: Create the migration resource if history is required -**Run on the workload cluster.** +Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required, and record that decision. Do not delete the source PVCs or PVs just to skip migration. -Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required. Record that decision. Do not delete the source PVCs or PVs just to skip migration. +If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. Creating it earlier is what records the final source state and the source volumes; a resource created after the uninstall has to name the source volumes explicitly and cannot use that final capture. -If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. Resource creation and the data copy happen at different times: the platform captures the final source state and records the source volumes when you create the resource, and copies the data only after the uninstall. Creating the migration after the uninstall requires explicit source volume references and cannot use the final capture. - -#### Target OpenSearch - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # Stable migration name, used by the commands below - namespace: cpaas-system -spec: - image: # Required; migration image provided for this release, with registry and tag or digest - source: - indexScope: # Required, selects the historical indices to migrate - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - target: - type: opensearch # Must match the PlatformLogForward target - secretRef: - name: platform-default-os-conn # The same connection Secret as PlatformLogForward - namespace: cpaas-system - options: - batchSize: 250 # Documents written per batch, 1~100000; default 250 - syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (no fixed wait for OpenSearch) - maxConcurrentJobs: 1 # Concurrent jobs; default 1, up to 2 for OpenSearch -``` - -#### Target ClickHouse - -Change `target.type` to `clickhouse` and point the Secret at `platform-default-ch-conn`. - -```yaml -apiVersion: log.alauda.io/v1alpha1 -kind: LegacyESMigration -metadata: - name: platform-es-history # Stable migration name, used by the commands below - namespace: cpaas-system -spec: - image: # Required; migration image provided for this release, with registry and tag or digest - source: - indexScope: # Required, selects the historical indices to migrate - - "log-workload-*" - - "log-platform-*" - - "log-system-*" - - "log-kubernetes-*" - - "event-*" - - "audit-*" - target: - type: clickhouse # Must match the PlatformLogForward target - secretRef: - name: platform-default-ch-conn # The same connection Secret as PlatformLogForward - namespace: cpaas-system - options: - batchSize: 250 # Documents written per batch, 1~100000; default 250 - syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (1 second for ClickHouse) - maxConcurrentJobs: 1 # Concurrent jobs; ClickHouse is always serial (1) -``` - -The standard `indexScope` categories are: - -| `indexScope` value | Data | Full index name example | -| --- | --- | --- | -| `log-workload-*` | Application and container logs | `log-workload-20260825` | -| `log-platform-*` | Platform component logs | `log-platform-20260825` | -| `log-system-*` | System logs | `log-system-20260825` | -| `log-kubernetes-*` | Kubernetes logs | `log-kubernetes-20260825` | -| `event-*` | Kubernetes events | `event-20260825` | -| `audit-*` | Audit logs | `audit-20260825` | - -If the source also contains per-project logs or metering data, add `log-project-*` or `meter-*` to `indexScope`; otherwise that data is not migrated. Do not use a single-day index such as `audit-20260825` unless you intentionally want to migrate only that day. The `source` and `target` sections cannot be changed after creation. The `options` block is optional; adjust it only when your migration plan requires it (the example keeps the default batch size and concurrency). - -Do not set `source.pvcRefs` in this flow; the platform records the discovered source volumes in `status.resolvedPvcRefs`. If the migration reports `SourcePVCsUnavailable`, stop and contact support; do not delete or edit the migration resource. - -Apply the resource and wait for `PrecaptureReady`: - -```bash -kubectl -n cpaas-system apply -f legacy-es-migration.yaml - -# Watch the phase; press Ctrl+C to stop -kubectl -n cpaas-system get legacyesmigration platform-es-history -w - -# Show the conditions if the phase does not advance -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' -``` - -In this flow, data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 3 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. - -The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. - -Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. +The scope the migration supports, the requirement that the migrated time range still fits the target retention, the manifests, and the verification steps are all in [Historical Data Migration](../migration/index.mdx). ### Step 3: Uninstall the legacy Elasticsearch storage plugin @@ -359,33 +265,7 @@ If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and r After the stability check passes, produce new log, event, and audit records and confirm that they can be queried from the new target. Do not continue to migration verification if the new data path is unhealthy. -### Step 4: Complete and verify the historical migration (optional) - -**Run on the workload cluster.** - -If you created `LegacyESMigration`, continue observing the same resource. Do not delete or recreate it. The phase should move from `PrecaptureReady` to `Validating`, `Running` (or `Retrying`), and finally `Succeeded`. - -```bash -kubectl -n cpaas-system get legacyesmigration platform-es-history -w - -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{.status.phase}{"\n"}{.status.progress}{"\n"}' -kubectl -n cpaas-system get legacyesmigration platform-es-history \ - -o jsonpath='{range .status.tasks[*]}{.pvcName}{"\t"}{.phase}{"\t"}{.processedDocs}{"\t"}{.lastError}{"\n"}{end}' -``` - -**Verification:** - -- `status.phase` is `Succeeded`. -- Every entry in `status.tasks` succeeded. -- Sample queries against the new target return the expected history for each `indexScope` category. -- The protected source PVCs and PVs are still present and bound. - -If the phase is `Blocked` or `Failed`, do not delete the migration resource, Jobs, or source volumes; contact support. - -If no migration was created, record the explicit decision that historical data is not required, and keep the protected source volumes until that decision is approved. - -### Step 5: Retained volumes +### Step 4: Retained volumes The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until migration and target validation are complete. Do not remove protection annotations, delete PVCs or PVs, or remove finalizers. To release the volumes after validation, contact Alauda support or follow the separate approved cleanup procedure. From 7b70bfc7005b56a7334a18f08c6644635cffbc7b Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 13:36:56 +0800 Subject: [PATCH 35/38] docs(zh): add the migration chapter and split the Chinese upgrade chapter Mirror the English split: the Chinese upgrade chapter now keeps only the upgrade steps and links to a new Historical Data Migration chapter that carries the LegacyESMigration manifests, the capture and copy sequence and the verification. Also add a warning to both migration chapters: the source volumes have to be provided in time. Discovery only works while the legacy StatefulSet and its PVCs are still present, and the operator never guesses a source volume, so the PVCs must be listed in source.pvcRefs, or the resource created, before anything deletes the claims or releases the protected volumes. Checked both pairs: identical code blocks after stripping comments, matching heading ranks, and each sourceSHA matching its English source. --- docs/en/migration/index.mdx | 4 + docs/zh/migration/index.mdx | 131 ++++++++++++++++ docs/zh/upgrade/index.mdx | 296 ++++++++++++++++++++++++++++++++++++ 3 files changed, 431 insertions(+) create mode 100644 docs/zh/migration/index.mdx create mode 100644 docs/zh/upgrade/index.mdx diff --git a/docs/en/migration/index.mdx b/docs/en/migration/index.mdx index 0e4dcd2..be93de8 100644 --- a/docs/en/migration/index.mdx +++ b/docs/en/migration/index.mdx @@ -17,6 +17,10 @@ This chapter migrates the historical log, event, and audit data that the legacy | Concurrency | ClickHouse targets always run one job at a time. OpenSearch targets accept `options.maxConcurrentJobs: 2`; the default is `1`. | | Batch size | `options.batchSize` accepts `1..100000`; `options.syncIntervalSeconds` sets the minimum wait between batches. | +:::warning +Provide the source volumes in time. Discovery only works while the legacy StatefulSet and its PVCs are still present. If they are already gone, or if discovery fails, list the PVCs yourself in `source.pvcRefs` — the operator never guesses a source volume. Once the source PVCs are deleted, or their PVs are released and cleaned up, the historical data can no longer be migrated, so create the migration resource, or supply `pvcRefs`, before anything releases those volumes. +::: + ## Before you start 1. The new data path exists and is `Ready`; its `PlatformLogForward` points at the target you migrate into. Follow the upgrade chapter through the step that installs the new data path first. diff --git a/docs/zh/migration/index.mdx b/docs/zh/migration/index.mdx new file mode 100644 index 0000000..727a5a7 --- /dev/null +++ b/docs/zh/migration/index.mdx @@ -0,0 +1,131 @@ +--- +weight: 16 +sourceSHA: 8319ef6824e22875580c08ab7b995c763d56035d14030db977d8b38ba8782826 +--- + +# 历史数据迁移 + +本章把旧 Elasticsearch 存储插件中的历史日志、事件与审计数据迁移到新数据链路使用的 ClickHouse 或 OpenSearch 目标端。它属于升级流程的一部分，升级章节会在适用位置链接到本章。 + +## 支持范围 + +| 项 | 支持情况 | +| --- | --- | +| 源 | 旧 Elasticsearch 存储插件的 PVC。不填写 `source.pvcRefs` 时，只要旧 StatefulSet 及其 PVC 还在，Operator 会自动发现;已经无法发现时（例如旧插件已卸载）必须显式列出。 | +| 数据 | 由 `source.indexScope` 选中的索引，支持精确索引名与 glob，必须至少命中一个索引;空列表会被拒绝。 | +| 目标 | `clickhouse` 或 `opensearch`。`target.secretRef` 复用与 `PlatformLogForward` 相同的连接 Secret。 | +| 时机 | 创建资源时抓取迁移边界，拷贝在旧插件卸载之后执行。 | +| 并发 | ClickHouse 目标始终单任务执行;OpenSearch 目标可设置 `options.maxConcurrentJobs: 2`，默认为 `1`。 | +| 批次 | `options.batchSize` 取值 `1..100000`;`options.syncIntervalSeconds` 控制批次之间的最小等待。 | + +:::warning +请在源卷被释放或删除之前及时提供源卷。自动发现只在旧 StatefulSet 及其 PVC 仍然存在时有效;如果它们已经不存在,或者发现失败,请在 `source.pvcRefs` 中自行列出这些 PVC —— Operator 不会猜测源卷。一旦源端 PVC 被删除、或其 PV 被释放并清理,历史数据就无法再迁移,因此请在相关卷被释放之前创建迁移资源或提供 `pvcRefs`。 +::: + +## 开始之前 + +1. 新数据链路已存在且为 `Ready`，其 `PlatformLogForward` 指向你要迁移到的目标端。请先按升级章节完成到"安装新数据链路"这一步。 +2. 本版本提供的迁移 worker 镜像可用，含完整 registry、tag 或 digest。`spec.image` 为必填;除非你更新它，Operator 会一直使用资源创建时的镜像。 +3. 目标端账号能够创建并写入迁移所需的表或索引，且与连接 Secret 里的是同一个账号。 +4. **迁移的时间范围必须仍在目标表的保留期内。** 迁移不会放宽、暂停或绕过 `PlatformLogForward` 与 razor 已生效的保留策略，因此早于目标 TTL 的记录可能写入后又被正常 merge 删除。请先确认时间范围，或先上调目标保留期并确认表定义已生效，再创建该资源。 + +## 步骤 1：创建迁移资源 + +请在卸载旧插件**之前**创建 `LegacyESMigration`。提前创建才会记录最终源状态与源端存储卷;卸载之后再创建，必须显式指定源卷，无法使用最终快照。 + +### 目标 OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # 固定的迁移名称，后续命令使用 + namespace: cpaas-system +spec: + image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest + source: + indexScope: # 必填，选择要迁移的历史索引 + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # 必须与 PlatformLogForward 的目标一致 + secretRef: + name: platform-default-os-conn # 与 PlatformLogForward 使用同一个连接 Secret + namespace: cpaas-system + options: + batchSize: 250 # 每批写入的文档数，1~100000；默认 250 + syncIntervalSeconds: 5 # 每批之间的最小等待时间；不填则使用默认值（OpenSearch 无固定等待） + maxConcurrentJobs: 1 # 并发任务数；默认 1，OpenSearch 最多 2 +``` + +### 目标 ClickHouse + +请把 `target.type` 改为 `clickhouse`，并把 Secret 指向 `platform-default-ch-conn`。 + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # 固定的迁移名称，后续命令使用 + namespace: cpaas-system +spec: + image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest + source: + indexScope: # 必填，选择要迁移的历史索引 + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # 必须与 PlatformLogForward 的目标一致 + secretRef: + name: platform-default-ch-conn # 与 PlatformLogForward 使用同一个连接 Secret + namespace: cpaas-system + options: + batchSize: 250 # 每批写入的文档数，1~100000；默认 250 + syncIntervalSeconds: 5 # 每批之间的最小等待时间；不填则使用默认值（ClickHouse 为 1 秒） + maxConcurrentJobs: 1 # 并发任务数；ClickHouse 始终串行（1） +``` + +请把 YAML 保存为 `legacy-es-migration.yaml` 并应用： + +```bash +kubectl apply -f legacy-es-migration.yaml +``` + +## 步骤 2：等待抓取完成 + +拷贝要等到旧插件卸载之后才开始，因此这一步的 phase 会先停在 `PrecaptureReady`： + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history -w +``` + +在本流程中，只有旧插件卸载之后才会开始复制数据，因此在此之前 phase 会一直停留在 `PrecaptureReady`。只有 phase 为 `PrecaptureReady`（已完成的迁移为 `Succeeded`）时才继续步骤 4。如果 phase 为 `Blocked`，请勿删除或重建迁移资源；请阅读 condition message 并联系支持。 + +迁移过程中新数据链路会持续接收日志、事件和审计数据。请保持新数据链路及其目标连接不变。如果迁移无法完成，请停止并联系支持。 + +请使用本 ACP 4.4 日志版本提供的迁移镜像。如果目标数据已经完整但迁移一直停留在 `Running`，请停止并联系支持；请勿删除迁移资源或源端存储卷。 + +## 步骤 3：校验迁移 + +旧插件卸载后，worker 会开始拷贝数据。观察同一个资源直到成功： + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}' # 期望：Succeeded +``` + +如果 phase 为 `Blocked`，请勿删除或重建该资源;请阅读 condition message 并联系支持。 + +随后在目标端确认迁移后的数据：用与生产相同的日志、事件与审计查询，确认历史时间范围的数据存在。 + +## 保留的存储卷 + +在校验通过、且变更窗口关闭之前，请保留受保护的源端存储卷。`status.phase: Succeeded` 本身不等于可以释放它们。 diff --git a/docs/zh/upgrade/index.mdx b/docs/zh/upgrade/index.mdx new file mode 100644 index 0000000..6c02847 --- /dev/null +++ b/docs/zh/upgrade/index.mdx @@ -0,0 +1,296 @@ +--- +weight: 15 +sourceSHA: 79a73a4ff41a9767de749a344e865068eee4a08408cd98c8aeac2ba091c3e0a9 +--- + +# 升级 + +本文介绍如何升级现有 ACP 部署中的 **Alauda Container Platform Log Storage for Elasticsearch**。 + +## 简介 + +本指南用于将使用 **Alauda Container Platform Log Storage for Elasticsearch** 存储日志的集群升级到 ACP 4.4。升级后，新的日志数据将写入 ClickHouse 或 OpenSearch 3.7.0 集群。 + +升级过程中 Elasticsearch 及其存储卷保持不动，平台会在旧链路旁边创建新链路，因此日志采集不会中断： + +1. 平台创建新的日志接收与存储链路，并开始将新的日志、事件和审计数据写入该链路。 +2. 旧 Elasticsearch 集群和旧链路保持运行，直到队列中的数据消费完成。 +3. 切换和排空完成后，PlatformLogForward 会报告 `LegacyESUpgradeCompleted`。 +4. 如果需要保留历史数据，请创建迁移资源，并在卸载旧插件前等待其进入 `PrecaptureReady`。 +5. 确认升级程序已完成旧 Elasticsearch 存储卷保护，然后卸载旧插件。 +6. 持续观察同一个迁移资源直到其成功，并在校验完成前保留受保护的源端存储卷。 + +:::warning +在本指南明确要求之前，请勿卸载 Elasticsearch 存储插件、停止旧数据链路，或删除其 PVC 和 PV。提前执行这些操作可能导致历史数据不可用，或导致无法捕获源端元数据快照。 +::: + +## 适用场景 + +当集群符合下面的源端和目标端条件时，请使用本指南。 + +| 项目 | 值 | +| --- | --- | +| 源端 | 已安装并运行 **Alauda Container Platform Log Storage for Elasticsearch** 的业务集群 | +| 目标平台 | ACP 4.4 | +| 目标存储 | 单独准备的 ClickHouse 或 OpenSearch 3.7.0 集群 | +| 支持的源端日志插件版本 | 4.2.x、4.3.x | + +## 前置条件 + +开始前，请确保： + +1. ACP 4.4 平台升级已完成，且日志组件可以升级。请在本流程中升级日志控制组件，并保持 **Alauda Container Platform Log Storage for Elasticsearch** 处于安装状态。 +2. 已从 **Alauda Cloud** 下载 ACP 4.4 日志插件包，且该插件包已上传到集群的插件市场。 +3. 已单独准备本次升级所需的目标存储和消息队列。本次升级不会复用 Elasticsearch 插件自带的存储和 Kafka，需要准备： + - 对于 OpenSearch 3.7.0，`analysis-ik` 插件可选。未安装时，日志查询会退回到不带中文分词的 standard 分析器：查询仍可正常执行，但中文全文检索效果会下降。如果现场需要中文检索质量，请在每个节点安装与 3.7.0 匹配的该插件。连接账号需要具备索引模板和生命周期策略的管理权限，以及日志数据的读写权限。 + - 对于 ClickHouse，请使用支持 `ReplicatedMergeTree` 和 Keeper/ZooKeeper 的复制集群。连接 Secret 中的 `cluster` 值必须与 ClickHouse 集群名一致；目标数据库必须在创建 PlatformLogForward 之前存在，如果新链路因 `UNKNOWN_DATABASE` 一直未就绪，请先创建数据库并等待下一次调谐。连接账号需要具备建表和改表、数据读写权限，以及执行 `SYSTEM DROP DNS CACHE` 的权限。 + - 新的 Kafka 服务，并已创建 `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC` 和 `ALAUDA_AUDIT_TOPIC` 三个 topic，日志 Kafka 用户已获得这些 topic 及相关消费组的访问权限。 + - 按各自产品文档准备好的目标存储和 Kafka 连接信息。 +4. 如果需要迁移历史数据，已取得本版本提供的迁移镜像，并包含完整的 registry、tag 或 digest。请勿复用旧版本的镜像。 +5. 已获得批准的变更窗口，且平台管理员可同时访问 global 管理集群和目标业务集群。在托管环境中，请与 Alauda 支持协同。 +6. 旧 Elasticsearch、Kafka、ZooKeeper、lanaya 和 Razor 工作负载仍在运行。在本指南要求之前，请勿停止、缩容或删除它们。 + +:::warning +升级期间，请勿停止、缩容或删除旧 Elasticsearch、Kafka、ZooKeeper、lanaya 或 Razor 工作负载；请勿删除 PVC、PV 或保护 finalizer；请勿删除或重建 `LegacyESMigration` 资源。这些操作可能导致历史数据不可用，或破坏迁移边界。 +::: + +## 目标存储准备 + +Operator 不会创建外部 OpenSearch/ClickHouse 集群和新的 Kafka 服务，需要单独创建，并通过步骤 1 中的 Secret 提供连接信息。 + +请参考[日志组件容量规划](https://docs.alauda.cn/logging-service/4.4/architecture/capacity_planning.html)规划容量，且不要低于当前 Alauda Container Platform Log Storage for Elasticsearch 部署的规格。目标存储需要能够容纳迁移的历史数据和新产生的数据。该指南中的参考磁盘配置为 `6000 IOPS`、`250 MB/s` 读写、独立 SSD 挂载；如果实际存储性能低于该配置，请选择更大规格。 + +表中列出的参数是当前部署的基线，不是目标端配置。现场实际值可能不同，规划前请先核实： + +| 旧部署参数 | 典型基线 | 需要为目标端准备什么 | +| --- | --- | --- | +| `elasticsearch.storage.node_size`、`node_replicas` | 每个数据节点 200 Gi；`node_replicas` 表示数据节点数量（默认 1，小规模档位为 3） | 按历史数据量、新增数据量和目标端 HA 策略规划目标存储 | +| `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | 目标端使用独立存储，不复用旧的本地路径 | +| `logging.esReplicas`、`logging.shards` | 索引级配置：1 副本、按类型设置分片数 | 仅适用于 OpenSearch：平台索引模板的起点是 1 分片、1 副本，不会自动沿用这些旧值；如果目标端需要不同的值，请覆盖模板。它们不适用于 ClickHouse——ClickHouse 使用 `externalStorage.shards` 和 `replicas` 描述集群拓扑（见下方对齐表） | +| `kafka.retention_hours` | 48 小时 | 在目标 `KafkaTopic` CR 中设置相同的保留时间；Kafka broker 的容量请按你的 Kafka 部署规划，不要沿用该旧参数 | +| `logging.ttl` | 日志 7 天；事件/审计 180 天；计量 540 天 | PlatformLogForward 会把相同的 TTL 下发到目标端；请按实际配置的保留时间规划容量 | + +Operator 会根据 PlatformLogForward 中的配置下发 TTL，以及 ClickHouse 的分片和副本数，因此准备好的集群必须能够满足这些值。这些参数不包含目标端节点规格：ClickHouse 请按容量规划的档位选择，OpenSearch 请按 OpenSearch 的部署规格并使用相同的数据量和吞吐输入进行规划。 + +### 与 PlatformLogForward 对齐目标端配置 + +请以环境中实际部署的 CR 为准读取目标端配置，不要以本文档中的值为准： + +| 目标端 | 从哪里读取实际部署的配置 | 需要对齐什么 | +| --- | --- | --- | +| ClickHouse | `ClickHouseInstallation` CR：`spec.configuration.clusters[].layout.shardsCount` 和 `replicasCount` | 将 PlatformLogForward 的 `spec.externalStorage.shards` 和 `replicas` 设置为 CHI CR 中声明的值，并将 Secret 中的 `cluster` 设置为 `ON CLUSTER` 使用的集群名。数据链路会按这两个值展开，配置不一致会写入错误的 replica set | +| Kafka | `Kafka` CR（`spec.kafka.config`）、`KafkaNodePool`（旧布局为 `Kafka.spec.kafka`）中的 broker 数量和存储、`KafkaTopic` CR、`KafkaUser` CR | Secret 中的 topic 名（`topics.log`、`topics.event`、`topics.audit`）必须与 `KafkaTopic.spec.topicName` 以及 `KafkaUser` ACL 中的 topic 名完全一致，`kafkaClusterName` 必须与 Kafka CR 名称一致。分区数、副本因子和保留时间在 `KafkaTopic` CR 中设置（或使用你的 Kafka 部署中对应的 topic 配置）；副本因子不能超过 broker 数量。Broker 和 topic 的最大消息大小必须能够容纳审计批次——Kafka 默认的 1 MiB 不够，参考 CR 使用 10 MiB。保持 `auto.create.topics.enable` 为关闭，避免自动创建名称错误的 topic | +| OpenSearch | 实际部署的 OpenSearch 集群（其 operator CR 或运行它的 manifest），以及已生效的索引模板（`GET /_index_template`） | 索引的分片和副本不由 PlatformLogForward 控制。平台会下发低优先级模板，默认为 1 分片、1 副本；如果生产环境的节点数或 HA 策略需要不同的值，请在切换前应用更高优先级的 composable template，并通过 `GET /_index_template` 确认结果。已创建的索引会保留创建时的设置 | + +## 升级前检查清单 + +| 检查项 | 期望结果 | +| --- | --- | +| 源端 | 旧 ES、Kafka、ZooKeeper、lanaya 和 Razor 均在运行；Alauda Container Platform Log Storage for Elasticsearch 为 `Running` | +| 目标存储 | OpenSearch 3.7.0 或受支持的 ClickHouse 拓扑可访问；数据库/Keeper 要求已满足 | +| 目标 Kafka | 新的 bootstrap 可访问；topic 和日志用户权限已就绪 | +| 连接 Secret | 必需 key 齐全且有效；未复用旧 Secret | +| 访问与变更窗口 | 平台管理员可访问 global 集群和业务集群；支持路径和变更窗口已确认 | +| 历史迁移 | 已取得本版本的迁移镜像；目标账号可以创建并写入所需的 schema 和数据 | + +## 升级流程概览 + +请按以下顺序执行。每步的卡点通过后才能进入下一步。 + +| 步骤 | 执行位置 | 操作 | 继续执行的条件 | +| --- | --- | --- | --- | +| 1 | 业务集群 | 安装新数据链路：连接 Secret + 带升级 annotation 的 `PlatformLogForward` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | +| 2 | 业务集群 | 如需历史数据，按[历史数据迁移](https://docs.alauda.cn/logging-service/4.3/migration/index.html)创建 `LegacyESMigration` | `PrecaptureReady`（已有迁移时为 `Succeeded`） | +| 3 | global 集群与业务集群 | 执行旧插件的受控卸载 | 60 秒内 `ModuleInfo`、`ClusterPluginInstance`、`AppRelease` 均未出现，且新日志查询正常 | +| 4 | 业务集群 | 观察同一个 `LegacyESMigration` | `Phase=Succeeded` 且目标端查询通过 | +| 5 | 业务集群 | 保留受保护的源端存储卷 | 清理前需获得明确批准 | + +## 升级步骤 + +### 步骤 1：带着升级 annotation 安装新数据链路 + +新数据链路的安装方式与全新安装完全相同：先创建目标存储和 Kafka 的连接 Secret，再创建一个 `PlatformLogForward`。相关清单与说明见[安装](https://docs.alauda.cn/logging-service/4.3/install/index.html)中的「外部 ClickHouse 或 OpenSearch 的日志存储安装」章节。 + +**唯一的区别是 annotation。** 请在 `PlatformLogForward` 上添加 `log.alauda.io/legacy-es-upgrade: "true"`，让平台进入旧 Elasticsearch 升级流程，而不是直接切换日志入口： + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 集群内固定的单例名称，请勿修改 + annotations: + log.alauda.io/legacy-es-upgrade: "true" # 固定值，进入旧 Elasticsearch 升级流程 +spec: + installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + # externalStorage 与 externalMessageQueue：同「安装」章节， + # 指向上面创建的目标存储与 Kafka 连接 Secret +``` + +请保持 `installMode: Fresh`，`Adopt` 不属于本流程。 + +新数据链路不会立即就绪。平台会先创建链路，然后切换日志入口，最后等待旧集群中排队的数据被消费完；耗时取决于积压量。请观察状态直到完成，按 `Ctrl+C` 结束： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 列变为 `Ready`，同时 `Ready` 列变为 `True`。只有看到 `Ready` 后才能继续。 + +如需跟踪进度或排查问题，可以查看状态 conditions： + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +请关注 `LegacyESUpgrade` 这一行：当 reason 变为 `LegacyESUpgradeCompleted` 时，说明日志入口已切换到新数据链路，且旧集群中排队的数据已消费完，本步骤完成。如果 reason 为 `Blocked`，`message` 会说明原因。 + +本步骤完成后，请生成或找到新的日志、事件和审计记录，确认可以从新目标端查询到，然后再继续。`LegacyESUpgrade` 未完成前，请勿卸载旧插件。 + +如果源端集群使用旧 Kafka，请勿停止或缩容 Kafka、ZooKeeper 或 lanaya。平台会自动通过旧链路排空排队数据，并在旧消费组 lag 归零时记录 `LegacyKafkaDrained`；`LegacyESUpgradeCompleted` 是本流程的卡点。 + +### 步骤 2：如需历史数据，创建迁移资源 + +只有在已批准的升级方案确认**不需要**历史 Elasticsearch 数据时才可跳过本步骤，并请记录该决定。不要为了跳过迁移而删除源端 PVC 或 PV。 + +如果需要这些数据，请在卸载旧插件**之前**创建 `LegacyESMigration`。提前创建才会记录最终源状态与源端存储卷;卸载之后再创建，必须显式指定源卷，无法使用最终快照。 + +迁移的支持范围、迁移时间范围必须落在目标保留期内的要求、清单与校验步骤，全部见[历史数据迁移](https://docs.alauda.cn/logging-service/4.3/migration/index.html)。 + +### 步骤 3：卸载旧 Elasticsearch 存储插件 + +:::warning +只有在以下条件全部满足时才能执行本步骤： + +- `PlatformLogForward/platform-default` 为 `Ready`，且其 `LegacyESUpgrade` condition 的 reason 为 `LegacyESUpgradeCompleted`。 +- 如需历史迁移，`LegacyESMigration/platform-es-history` 为 `PrecaptureReady` 或 `Succeeded`。 +- 请确认升级程序已完成旧 Elasticsearch 存储卷（PVC/PV）的保护。请勿修改或移除该保护；如果平台报告保护失败，请停止并联系支持。 +- 如果源端使用旧 Kafka，平台已确认所有排队数据均已排空。请勿通过停止或缩容 Kafka、ZooKeeper、lanaya 或 Elasticsearch 来强制达到该状态。 +::: + +:::warning +本流程会临时清理 Alauda Container Platform Log Storage for Elasticsearch 的平台发现字段，以便卸载旧插件。请仅在批准的变更窗口内执行，并请勿修改或删除 logagent 及其依赖。 +::: + +如果平台提供了本次升级支持的插件卸载操作，请先按平台或支持人员的说明执行。如果该操作被拒绝，或平台团队要求执行受控流程，请使用下面的步骤。 + +两次删除必须连续执行：中间不要等待 `ModuleInfo` 消失、不要检查 `AppRelease`，也不要执行其他检查。 + +**在 global 集群：开始关键步骤** + +将 `CLUSTER` 设置为业务集群在 global 集群中注册的名称，然后执行： + +```bash +set -euo pipefail + +CLUSTER= + +# 1. 解析目标 ModuleInfo。结果不唯一时停止。 +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" +if [ "$MODULE_COUNT" -gt 1 ]; then + echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 + exit 1 +fi + +if [ "$MODULE_COUNT" -eq 1 ]; then + MODULE_INFO="$MODULE_INFOS" + MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" + MODULE_CONFIG="logcenter-${MODULE_VERSION}" + + # 2. 修改前先备份当前的管理对象。 + kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml + kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml + kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml + + # 3. 清理发现标记，这一步同时绕过 logagent 的依赖检查。 + kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + + # 4. 删除请求被接受后立即返回，不要在此等待。 + kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false +else + echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." +fi +``` + +**在业务集群：完成关键步骤** + +立即把 kubectl context 切换到业务集群，不要先执行任何等待或校验。执行： + +```bash +# 5. 删除单集群安装记录，避免其重新创建 ModuleInfo。 +kubectl delete clusterplugininstance logcenter --ignore-not-found +``` + +**等待旧数据链路移除** + +切回 global 集群，等待 `ModuleInfo` 消失： + +```bash +CLUSTER= +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" + +if [ -n "$MODULE_INFO" ]; then + kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No ModuleInfo remains for cluster $CLUSTER." +fi +``` + +切换到业务集群，等待旧 `AppRelease` 被删除： + +```bash +kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +完成这些步骤后，等待 60 秒并确认这些资源没有重新出现。每条命令都不应返回资源。 + +**在 global 集群** + +```bash +kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found +``` + +**在业务集群** + +```bash +kubectl get clusterplugininstance logcenter --ignore-not-found +kubectl -n cpaas-system get apprelease logcenter --ignore-not-found +kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found +``` + +如果 `ClusterPluginInstance/logcenter` 仍然存在或重新出现，请再次删除并重复检查；只要它存在，平台就可能重新创建 `ModuleInfo`。如果 `ModuleInfo` 也重新出现，请先删除 `ClusterPluginInstance/logcenter`，再删除新的 `ModuleInfo`。请勿手动恢复被清理的发现字段；只需确认 `ModuleInfo`、`ClusterPluginInstance` 和 `AppRelease` 不再出现。如果 global 集群的步骤在删除 `ModuleInfo` 之前失败，请勿执行本次的业务集群步骤：先修复原因，再重新执行 global 集群步骤。 + +稳定性检查通过后，生成新的日志、事件和审计记录，并确认可以从新目标端查询到。如果新数据链路不健康，请勿继续迁移校验。 + +### 步骤 4：保留的存储卷 + +受保护的旧 Elasticsearch PVC 和 PV 会被保留。请在迁移和目标端校验完成前保留它们。请勿移除保护注解、删除 PVC/PV 或移除 finalizer。校验完成后如需释放这些存储卷，请联系 Alauda 支持或按照单独的、已批准的清理流程执行。 + +本次升级只保留 ES 的 PVC 和 PV。旧 Kafka 和 ZooKeeper 的存储卷不在保留范围内。 + +请导出或记录最终的 PLF conditions、迁移状态、目标端查询结果以及保留的 PVC/PV 名称，用于实施交接。请勿在本次升级中移除保护注解或删除这些存储卷。 + +## 完成检查清单 + +| 状态 | 期望 | +| --- | --- | +| `PlatformLogForward/platform-default` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | +| `LegacyESMigration/platform-es-history`（如已创建） | `Phase=Succeeded` | +| 新日志查询 | 新的日志、事件和审计记录可以通过新查询链路返回 | +| 旧插件卸载 | `ModuleInfo`、`ClusterPluginInstance/logcenter`、`AppRelease/logcenter` 均不存在，旧 ES 工作负载已消失 | +| 源端存储卷 | 受保护的旧 ES PVC 和 PV 仍然存在 | + +如果任何状态不符合预期，请停止并联系 Alauda 支持。请勿通过删除迁移资源、源端存储卷或目标端数据来绕过故障。 + +## 步骤受阻时 + +- 平台以 `moduleinfo is depended by ...` 拒绝卸载插件：请停止，不要修改 logagent，重新执行受控流程；如果仍然失败，请联系支持。 +- 迁移为 `Blocked`、`Failed` 或一直停留在 `Running`：请勿删除迁移资源、Job、源端存储卷或目标端数据；请联系支持。 +- `ModuleInfo` 或 `ClusterPluginInstance` 重新出现：请先删除 `ClusterPluginInstance`，再删除新的 `ModuleInfo`，并重复 60 秒稳定性检查。 +- `AppRelease/logcenter` 未消失：请勿移除 finalizer；请联系支持。 +- 卸载后新日志查询失败：请停止并联系支持。请勿删除保留的存储卷。 From eca8314789fa9d3ef5ab1a3f6c12f8939517e773 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 13:42:39 +0800 Subject: [PATCH 36/38] docs: trim the upgrade and migration preamble Both chapters opened with sections that repeated each other and the procedure: an introduction that narrated the steps, a scenarios table, a target preparation section, an alignment table and a preflight checklist all restated the same preconditions before the reader reached the first actionable step. Keep what the reader cannot infer and drop the rest: - Upgrade keeps one short introduction, one warning about not touching the legacy workloads or volumes, a four-item Before you start that points at the preparation chapter, one compact alignment table (the only place the shards, topics and index-template constraints are stated), the flow table and the steps. - Migration keeps the supported scope table and the source volume warning, and its Before you start shrinks to the two facts that matter: the new data path has to be ready with the migration image available, and the migrated range has to fit inside the target retention. English and Chinese are trimmed the same way; both pairs keep identical code blocks, matching heading ranks and matching sourceSHA values. --- docs/en/migration/index.mdx | 6 +-- docs/en/upgrade/index.mdx | 92 +++++------------------------------- docs/zh/migration/index.mdx | 8 ++-- docs/zh/upgrade/index.mdx | 93 +++++-------------------------------- 4 files changed, 28 insertions(+), 171 deletions(-) diff --git a/docs/en/migration/index.mdx b/docs/en/migration/index.mdx index be93de8..fc4100e 100644 --- a/docs/en/migration/index.mdx +++ b/docs/en/migration/index.mdx @@ -23,10 +23,8 @@ Provide the source volumes in time. Discovery only works while the legacy Statef ## Before you start -1. The new data path exists and is `Ready`; its `PlatformLogForward` points at the target you migrate into. Follow the upgrade chapter through the step that installs the new data path first. -2. The migration worker image for this release is available, with its complete registry, tag, or digest. `spec.image` is required; the operator keeps using the image the resource was created with unless you update it. -3. The target account can create and write the tables or indices the migration needs, and it is the same account the connection Secret carries. -4. **The migrated time range is still inside the retention of the target tables.** The migration does not widen, pause, or bypass the retention that the `PlatformLogForward` and razor have already applied, so records older than the target TTL can be written and then removed by the normal merge. Confirm the range first, or raise the target retention and confirm the table definitions took effect before you create this resource. +1. The new data path is `Ready` and its `PlatformLogForward` points at the target you migrate into. The migration image for this release is available, with its complete registry, tag, or digest (`spec.image` is required). +2. **The migrated time range is still inside the retention of the target tables.** The migration does not widen, pause, or bypass the retention that the `PlatformLogForward` and razor already applied, so records older than the target TTL can be written and then removed by the normal merge. Confirm the range first, or raise the target retention and confirm the table definitions took effect, before you create this resource. ## Step 1: Create the migration resource diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx index 1fbe187..e1b20bd 100644 --- a/docs/en/upgrade/index.mdx +++ b/docs/en/upgrade/index.mdx @@ -6,92 +6,22 @@ weight: 15 This section explains how to upgrade **Alauda Container Platform Log Storage for Elasticsearch** in an existing ACP deployment. -## Introduction - -This guide upgrades a cluster that stores logs with **Alauda Container Platform Log Storage for Elasticsearch** to ACP 4.4, where new log data is written to ClickHouse or to an OpenSearch 3.7.0 cluster. - -Elasticsearch and its volumes stay in place while the platform creates the new path alongside the legacy one, so log collection is not interrupted: - -1. The platform creates the new log receiving and storage path and starts writing new log, event, and audit data to it. -2. The legacy Elasticsearch cluster and the old pipeline stay running while queued data is consumed. -3. The PlatformLogForward reports `LegacyESUpgradeCompleted` after cutover and drain complete. -4. If you need the historical data, create the migration resource and wait for `PrecaptureReady` before uninstalling the legacy plugin. -5. Confirm that the upgrade program has protected the legacy Elasticsearch volumes, then uninstall the plugin. -6. Observe the same migration resource until it succeeds, and keep the protected source volumes until validation is complete. - -:::warning -Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs and PVs before this guide tells you to. Doing it earlier can make the historical data unavailable, or prevent the source metadata snapshot from being captured. -::: - -## Scenarios - -Use this guide when the cluster matches the source and target below. - -| Item | Value | -| --- | --- | -| Source | A workload cluster with **Alauda Container Platform Log Storage for Elasticsearch** installed and running | -| Target platform | ACP 4.4 | -| Target storage | A separately prepared ClickHouse or OpenSearch 3.7.0 cluster | -| Supported source Logging plugin versions | 4.2.x, 4.3.x | - -## Prerequisites - -Before you start, ensure that: - -1. The ACP 4.4 platform upgrade is complete and the Logging components can be upgraded. Upgrade the Logging control components as part of this procedure, and keep **Alauda Container Platform Log Storage for Elasticsearch** installed. -2. You have downloaded the ACP 4.4 Logging plugin package from **Alauda Cloud**, and the package is available in the plugin marketplace of the cluster. -3. You have separately prepared the target storage and message queue used by this upgrade. The upgrade does not reuse the storage or Kafka that ship with the Elasticsearch plugin, so you provide: - - For OpenSearch 3.7.0, `analysis-ik` is optional. Without it, log searches fall back to a standard analyzer without Chinese word segmentation: they still work, but Chinese full-text search quality is lower. Install the matching 3.7.0 plugin on every node if that matters for the site. The connection account must be able to manage index templates and lifecycle policies and to read and write the log data. - - For ClickHouse, use a supported replicated cluster (`ReplicatedMergeTree` with Keeper/ZooKeeper). The `cluster` value in the connection Secret must match the ClickHouse cluster name, and the target database must exist before you create the PlatformLogForward; if the new data path stays unready with `UNKNOWN_DATABASE`, create the database and wait for the next reconciliation. The connection account must be able to create and update the schema, to read and write data, and to run `SYSTEM DROP DNS CACHE`. - - A new Kafka service, with the `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, and `ALAUDA_AUDIT_TOPIC` topics created, and the Logging Kafka user granted access to these topics and the related consumer groups. - - The target storage and Kafka connection details prepared as described in [Environment Preparation](../prepare/index.mdx). -4. If you migrate historical data, you have the migration image provided for this release, with the complete registry, tag, or digest. Do not reuse an image from an earlier version. -5. An approved change window is available, and a platform administrator can access both the global management cluster and the target workload cluster. In a managed environment, coordinate with Alauda support. -6. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running. Do not stop, scale, or delete them before this guide tells you to. - - :::warning -During the upgrade, do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Do not delete PVCs, PVs, or protection finalizers. Do not delete or recreate a `LegacyESMigration` resource. These actions can make historical data unavailable or invalidate the migration boundary. +Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs, PVs, or protection finalizers before this guide tells you to, and do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Doing it earlier can make the historical data unavailable or invalidate the migration boundary. ::: -## Target Storage Preparation - -The operator does not create the external OpenSearch/ClickHouse cluster or the new Kafka service. Create them as described in [Environment Preparation](../prepare/index.mdx), then provide the connection details through the Secrets in Step 1. - -Plan the size with [Log Component Capacity Planning](../architecture/capacity_planning.mdx), and do not plan below the current Alauda Container Platform Log Storage for Elasticsearch deployment. The target must hold the migrated history plus new traffic. The reference disk configuration in that guide is `6000 IOPS` and `250 MB/s` read/write on dedicated SSD mounts; if the actual storage is weaker, move to a larger profile. - -The values in the table are the current deployment baseline, not the target configuration. Actual site values may differ, so verify them before planning: +## Before you start -| Legacy deployment setting | Typical baseline | What to prepare for the target | -| --- | --- | --- | -| `elasticsearch.storage.node_size`, `node_replicas` | 200 Gi per data node; `node_replicas` sets the data-node count (1 by default; the small-scale profile uses 3) | Size the target storage for the historical data plus new traffic and the target HA policy | -| `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | Use independent storage for the new target; the legacy local path is not reused | -| `logging.esReplicas`, `logging.shards` | Index-level settings: 1 replica, per-type shard counts | OpenSearch only: the platform's index templates start at 1 shard and 1 replica and do not inherit these legacy values; override the template if the target needs different values. They do not map to ClickHouse — ClickHouse uses `externalStorage.shards` and `replicas` for the cluster topology (see the alignment table below) | -| `kafka.retention_hours` | 48 hours | Set the same retention in the target `KafkaTopic` CRs; size the Kafka brokers through your Kafka deployment, not from this legacy value | -| `logging.ttl` | Logs 7 days; events/audits 180 days; metering 540 days | `PlatformLogForward` applies the same TTLs to the target; plan capacity with the retention you configure there | +1. The target ClickHouse or OpenSearch cluster and the new Kafka service are prepared as described in [Environment Preparation](../prepare/index.mdx), and their connection details are available. +2. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running, and an approved change window is in place. +3. To migrate historical data, the migration image for this release is available, with its complete registry, tag, or digest. +4. The target matches what the `PlatformLogForward` will declare. Read the deployed CRs rather than this guide: -The operator applies the TTL and, for ClickHouse, the shard and replica values from the `PlatformLogForward` spec, so the prepared cluster must be able to satisfy them. These settings do not cover the target node size: size ClickHouse with the capacity-planning profiles, and size OpenSearch with your OpenSearch deployment sizing using the same data volume and throughput inputs. - -### Aligning the target with the PlatformLogForward - -Read the target settings from the CRs that are actually deployed in the environment, not from this guide: - -| Target | Where to read the deployed settings | What must line up | -| --- | --- | --- | -| ClickHouse | `ClickHouseInstallation` CR: `spec.configuration.clusters[].layout.shardsCount` and `replicasCount` | Set `spec.externalStorage.shards` and `replicas` in the `PlatformLogForward` to the values declared in the CHI CR, and set the Secret `cluster` value to the cluster name used by `ON CLUSTER`. The data path expands by these two values, so a mismatch writes to the wrong replica set. | -| Kafka | `Kafka` CR (`spec.kafka.config`), `KafkaNodePool` (or `Kafka.spec.kafka` in older layouts) for broker count and storage, `KafkaTopic` CRs, and `KafkaUser` CR | Keep the Secret topic names (`topics.log`, `topics.event`, `topics.audit`) equal to `KafkaTopic.spec.topicName` and to the topic names in the `KafkaUser` ACLs, and keep `kafkaClusterName` equal to the Kafka CR name. Set partitions, replication factor, and retention in the `KafkaTopic` CRs (or the equivalent topic configuration on your Kafka deployment); the replication factor cannot exceed the number of brokers. The broker and topic maximum message size must accept the audit batches — the Kafka default is 1 MiB, which is too small; the reference CRs use 10 MiB. Keep `auto.create.topics.enable` disabled so a misnamed topic is not created automatically. | -| OpenSearch | The deployed OpenSearch cluster (its operator CR or the manifests that run it) and the applied index templates (`GET /_index_template`) | Index shards and replicas are not controlled by the `PlatformLogForward`. The platform applies low-priority templates with 1 shard and 1 replica; if the production node count or HA policy needs different values, apply a higher-priority composable template before cutover and confirm the result with `GET /_index_template`. Existing indices keep the settings they were created with. | - -## Preflight Checklist - -| Check | Expected | -| --- | --- | -| Source | Legacy ES, Kafka, ZooKeeper, lanaya, and Razor are running; Alauda Container Platform Log Storage for Elasticsearch is `Running` | -| Target storage | OpenSearch 3.7.0 or the supported ClickHouse topology is reachable; database/Keeper requirements are met | -| Target Kafka | New bootstrap is reachable; topics and Logging user access are ready | -| Connection Secrets | Required keys are present and valid; legacy Secrets are not reused | -| Access and change window | Platform administrator can reach global and workload clusters; support path and change window are confirmed | -| Historical migration | The migration image for this release is available; the target account can create and write the required schema and data | + | Target | What must line up | + | --- | --- | + | ClickHouse | Set `externalStorage.shards` and `replicas` from the `ClickHouseInstallation` (`spec.configuration.clusters[].layout`), and set the Secret `cluster` value to the cluster name used by `ON CLUSTER`. A mismatch writes to the wrong replica set. | + | Kafka | Keep the Secret `topics.log` / `topics.event` / `topics.audit` equal to the deployed topic names and to the `KafkaUser` ACLs, and `kafkaClusterName` equal to the Kafka cluster name. The broker and topic maximum message size must accept the audit batches. | + | OpenSearch | Index shards and replicas are not controlled by the `PlatformLogForward`. The platform applies templates with 1 shard and 1 replica; apply a higher-priority template before cutover if production needs different values. | ## Upgrade Flow at a Glance diff --git a/docs/zh/migration/index.mdx b/docs/zh/migration/index.mdx index 727a5a7..a189492 100644 --- a/docs/zh/migration/index.mdx +++ b/docs/zh/migration/index.mdx @@ -1,6 +1,6 @@ --- weight: 16 -sourceSHA: 8319ef6824e22875580c08ab7b995c763d56035d14030db977d8b38ba8782826 +sourceSHA: ec0c45131648e842846c190ed191d1ca5089a33021d8263ca0a7a3433d3a780d --- # 历史数据迁移 @@ -24,10 +24,8 @@ sourceSHA: 8319ef6824e22875580c08ab7b995c763d56035d14030db977d8b38ba8782826 ## 开始之前 -1. 新数据链路已存在且为 `Ready`，其 `PlatformLogForward` 指向你要迁移到的目标端。请先按升级章节完成到"安装新数据链路"这一步。 -2. 本版本提供的迁移 worker 镜像可用，含完整 registry、tag 或 digest。`spec.image` 为必填;除非你更新它，Operator 会一直使用资源创建时的镜像。 -3. 目标端账号能够创建并写入迁移所需的表或索引，且与连接 Secret 里的是同一个账号。 -4. **迁移的时间范围必须仍在目标表的保留期内。** 迁移不会放宽、暂停或绕过 `PlatformLogForward` 与 razor 已生效的保留策略，因此早于目标 TTL 的记录可能写入后又被正常 merge 删除。请先确认时间范围，或先上调目标保留期并确认表定义已生效，再创建该资源。 +1. 新数据链路为 `Ready`，其 `PlatformLogForward` 指向你要迁移到的目标端;本版本的迁移镜像可用，并含完整的 registry、tag 或 digest（`spec.image` 为必填）。 +2. **迁移的时间范围必须仍在目标表的保留期内。** 迁移不会放宽、暂停或绕过 `PlatformLogForward` 与 razor 已生效的保留策略，因此早于目标 TTL 的记录可能写入后又被正常 merge 删除。请在创建该资源之前确认时间范围，或先上调目标保留期并确认表定义已生效。 ## 步骤 1：创建迁移资源 diff --git a/docs/zh/upgrade/index.mdx b/docs/zh/upgrade/index.mdx index 6c02847..51190b9 100644 --- a/docs/zh/upgrade/index.mdx +++ b/docs/zh/upgrade/index.mdx @@ -1,97 +1,28 @@ --- weight: 15 -sourceSHA: 79a73a4ff41a9767de749a344e865068eee4a08408cd98c8aeac2ba091c3e0a9 +sourceSHA: 937bc8eb61d1deab0bedea99d3237248087d3995447ea54692289f2dbaa3a2b4 --- # 升级 本文介绍如何升级现有 ACP 部署中的 **Alauda Container Platform Log Storage for Elasticsearch**。 -## 简介 - -本指南用于将使用 **Alauda Container Platform Log Storage for Elasticsearch** 存储日志的集群升级到 ACP 4.4。升级后，新的日志数据将写入 ClickHouse 或 OpenSearch 3.7.0 集群。 - -升级过程中 Elasticsearch 及其存储卷保持不动，平台会在旧链路旁边创建新链路，因此日志采集不会中断： - -1. 平台创建新的日志接收与存储链路，并开始将新的日志、事件和审计数据写入该链路。 -2. 旧 Elasticsearch 集群和旧链路保持运行，直到队列中的数据消费完成。 -3. 切换和排空完成后，PlatformLogForward 会报告 `LegacyESUpgradeCompleted`。 -4. 如果需要保留历史数据，请创建迁移资源，并在卸载旧插件前等待其进入 `PrecaptureReady`。 -5. 确认升级程序已完成旧 Elasticsearch 存储卷保护，然后卸载旧插件。 -6. 持续观察同一个迁移资源直到其成功，并在校验完成前保留受保护的源端存储卷。 - -:::warning -在本指南明确要求之前，请勿卸载 Elasticsearch 存储插件、停止旧数据链路，或删除其 PVC 和 PV。提前执行这些操作可能导致历史数据不可用，或导致无法捕获源端元数据快照。 -::: - -## 适用场景 - -当集群符合下面的源端和目标端条件时，请使用本指南。 - -| 项目 | 值 | -| --- | --- | -| 源端 | 已安装并运行 **Alauda Container Platform Log Storage for Elasticsearch** 的业务集群 | -| 目标平台 | ACP 4.4 | -| 目标存储 | 单独准备的 ClickHouse 或 OpenSearch 3.7.0 集群 | -| 支持的源端日志插件版本 | 4.2.x、4.3.x | - -## 前置条件 - -开始前，请确保： - -1. ACP 4.4 平台升级已完成，且日志组件可以升级。请在本流程中升级日志控制组件，并保持 **Alauda Container Platform Log Storage for Elasticsearch** 处于安装状态。 -2. 已从 **Alauda Cloud** 下载 ACP 4.4 日志插件包，且该插件包已上传到集群的插件市场。 -3. 已单独准备本次升级所需的目标存储和消息队列。本次升级不会复用 Elasticsearch 插件自带的存储和 Kafka，需要准备： - - 对于 OpenSearch 3.7.0，`analysis-ik` 插件可选。未安装时，日志查询会退回到不带中文分词的 standard 分析器：查询仍可正常执行，但中文全文检索效果会下降。如果现场需要中文检索质量，请在每个节点安装与 3.7.0 匹配的该插件。连接账号需要具备索引模板和生命周期策略的管理权限，以及日志数据的读写权限。 - - 对于 ClickHouse，请使用支持 `ReplicatedMergeTree` 和 Keeper/ZooKeeper 的复制集群。连接 Secret 中的 `cluster` 值必须与 ClickHouse 集群名一致；目标数据库必须在创建 PlatformLogForward 之前存在，如果新链路因 `UNKNOWN_DATABASE` 一直未就绪，请先创建数据库并等待下一次调谐。连接账号需要具备建表和改表、数据读写权限，以及执行 `SYSTEM DROP DNS CACHE` 的权限。 - - 新的 Kafka 服务，并已创建 `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC` 和 `ALAUDA_AUDIT_TOPIC` 三个 topic，日志 Kafka 用户已获得这些 topic 及相关消费组的访问权限。 - - 按各自产品文档准备好的目标存储和 Kafka 连接信息。 -4. 如果需要迁移历史数据，已取得本版本提供的迁移镜像，并包含完整的 registry、tag 或 digest。请勿复用旧版本的镜像。 -5. 已获得批准的变更窗口，且平台管理员可同时访问 global 管理集群和目标业务集群。在托管环境中，请与 Alauda 支持协同。 -6. 旧 Elasticsearch、Kafka、ZooKeeper、lanaya 和 Razor 工作负载仍在运行。在本指南要求之前，请勿停止、缩容或删除它们。 - :::warning -升级期间，请勿停止、缩容或删除旧 Elasticsearch、Kafka、ZooKeeper、lanaya 或 Razor 工作负载；请勿删除 PVC、PV 或保护 finalizer；请勿删除或重建 `LegacyESMigration` 资源。这些操作可能导致历史数据不可用，或破坏迁移边界。 +在本文档明确要求之前，请勿卸载 Elasticsearch 存储插件、停止旧数据链路，也不要删除其 PVC、PV 或保护 finalizer;同时请勿停止、缩容或删除旧的 Elasticsearch、Kafka、ZooKeeper、lanaya 与 Razor 工作负载。提前操作可能导致历史数据不可用，或破坏迁移边界。 ::: -## 目标存储准备 - -Operator 不会创建外部 OpenSearch/ClickHouse 集群和新的 Kafka 服务，需要单独创建，并通过步骤 1 中的 Secret 提供连接信息。 - -请参考[日志组件容量规划](https://docs.alauda.cn/logging-service/4.4/architecture/capacity_planning.html)规划容量，且不要低于当前 Alauda Container Platform Log Storage for Elasticsearch 部署的规格。目标存储需要能够容纳迁移的历史数据和新产生的数据。该指南中的参考磁盘配置为 `6000 IOPS`、`250 MB/s` 读写、独立 SSD 挂载；如果实际存储性能低于该配置，请选择更大规格。 - -表中列出的参数是当前部署的基线，不是目标端配置。现场实际值可能不同，规划前请先核实： - -| 旧部署参数 | 典型基线 | 需要为目标端准备什么 | -| --- | --- | --- | -| `elasticsearch.storage.node_size`、`node_replicas` | 每个数据节点 200 Gi；`node_replicas` 表示数据节点数量（默认 1，小规模档位为 3） | 按历史数据量、新增数据量和目标端 HA 策略规划目标存储 | -| `elasticsearch.hostpath` | `/cpaas/data/elasticsearch` | 目标端使用独立存储，不复用旧的本地路径 | -| `logging.esReplicas`、`logging.shards` | 索引级配置：1 副本、按类型设置分片数 | 仅适用于 OpenSearch：平台索引模板的起点是 1 分片、1 副本，不会自动沿用这些旧值；如果目标端需要不同的值，请覆盖模板。它们不适用于 ClickHouse——ClickHouse 使用 `externalStorage.shards` 和 `replicas` 描述集群拓扑（见下方对齐表） | -| `kafka.retention_hours` | 48 小时 | 在目标 `KafkaTopic` CR 中设置相同的保留时间；Kafka broker 的容量请按你的 Kafka 部署规划，不要沿用该旧参数 | -| `logging.ttl` | 日志 7 天；事件/审计 180 天；计量 540 天 | PlatformLogForward 会把相同的 TTL 下发到目标端；请按实际配置的保留时间规划容量 | +## 开始之前 -Operator 会根据 PlatformLogForward 中的配置下发 TTL，以及 ClickHouse 的分片和副本数，因此准备好的集群必须能够满足这些值。这些参数不包含目标端节点规格：ClickHouse 请按容量规划的档位选择，OpenSearch 请按 OpenSearch 的部署规格并使用相同的数据量和吞吐输入进行规划。 +1. 目标 ClickHouse 或 OpenSearch 集群以及新的 Kafka 服务已按[环境准备](https://docs.alauda.cn/logging-service/4.3/prepare/index.html)准备好，连接信息可用。 +2. 旧的 Elasticsearch、Kafka、ZooKeeper、lanaya 与 Razor 工作负载仍在运行，且已安排好变更窗口。 +3. 如需迁移历史数据，本版本的迁移镜像可用，并含完整的 registry、tag 或 digest。 +4. 目标端要与 `PlatformLogForward` 将要声明的取值一致。请以环境中实际部署的 CR 为准，而不是本文档： -### 与 PlatformLogForward 对齐目标端配置 - -请以环境中实际部署的 CR 为准读取目标端配置，不要以本文档中的值为准： - -| 目标端 | 从哪里读取实际部署的配置 | 需要对齐什么 | -| --- | --- | --- | -| ClickHouse | `ClickHouseInstallation` CR：`spec.configuration.clusters[].layout.shardsCount` 和 `replicasCount` | 将 PlatformLogForward 的 `spec.externalStorage.shards` 和 `replicas` 设置为 CHI CR 中声明的值，并将 Secret 中的 `cluster` 设置为 `ON CLUSTER` 使用的集群名。数据链路会按这两个值展开，配置不一致会写入错误的 replica set | -| Kafka | `Kafka` CR（`spec.kafka.config`）、`KafkaNodePool`（旧布局为 `Kafka.spec.kafka`）中的 broker 数量和存储、`KafkaTopic` CR、`KafkaUser` CR | Secret 中的 topic 名（`topics.log`、`topics.event`、`topics.audit`）必须与 `KafkaTopic.spec.topicName` 以及 `KafkaUser` ACL 中的 topic 名完全一致，`kafkaClusterName` 必须与 Kafka CR 名称一致。分区数、副本因子和保留时间在 `KafkaTopic` CR 中设置（或使用你的 Kafka 部署中对应的 topic 配置）；副本因子不能超过 broker 数量。Broker 和 topic 的最大消息大小必须能够容纳审计批次——Kafka 默认的 1 MiB 不够，参考 CR 使用 10 MiB。保持 `auto.create.topics.enable` 为关闭，避免自动创建名称错误的 topic | -| OpenSearch | 实际部署的 OpenSearch 集群（其 operator CR 或运行它的 manifest），以及已生效的索引模板（`GET /_index_template`） | 索引的分片和副本不由 PlatformLogForward 控制。平台会下发低优先级模板，默认为 1 分片、1 副本；如果生产环境的节点数或 HA 策略需要不同的值，请在切换前应用更高优先级的 composable template，并通过 `GET /_index_template` 确认结果。已创建的索引会保留创建时的设置 | - -## 升级前检查清单 - -| 检查项 | 期望结果 | -| --- | --- | -| 源端 | 旧 ES、Kafka、ZooKeeper、lanaya 和 Razor 均在运行；Alauda Container Platform Log Storage for Elasticsearch 为 `Running` | -| 目标存储 | OpenSearch 3.7.0 或受支持的 ClickHouse 拓扑可访问；数据库/Keeper 要求已满足 | -| 目标 Kafka | 新的 bootstrap 可访问；topic 和日志用户权限已就绪 | -| 连接 Secret | 必需 key 齐全且有效；未复用旧 Secret | -| 访问与变更窗口 | 平台管理员可访问 global 集群和业务集群；支持路径和变更窗口已确认 | -| 历史迁移 | 已取得本版本的迁移镜像；目标账号可以创建并写入所需的 schema 和数据 | + | 目标 | 必须对齐的项 | + | --- | --- | + | ClickHouse | `externalStorage.shards` 与 `replicas` 按 `ClickHouseInstallation`（`spec.configuration.clusters[].layout`）填写;Secret 的 `cluster` 取 `ON CLUSTER` 使用的集群名。不一致会把数据写到错误的副本集。 | + | Kafka | Secret 的 `topics.log` / `topics.event` / `topics.audit` 必须与实际 topic 名及 `KafkaUser` 的 ACL 一致，`kafkaClusterName` 与 Kafka 集群名一致;broker 与 topic 的最大消息大小必须能容纳审计批次。 | + | OpenSearch | 索引的分片与副本不由 `PlatformLogForward` 控制。平台下发的是 1 分片 1 副本的模板;如果生产需要不同取值，请在切换前下发更高优先级的模板。 | ## 升级流程概览 From c356e570a789a18218219ee4fecfcc56b43f51ea Mon Sep 17 00:00:00 2001 From: root Date: Fri, 18 Sep 2026 13:52:42 +0800 Subject: [PATCH 37/38] docs: prepare and install external logging storage - prepare: split ClickHouse and OpenSearch, single-node and multi-node, and make every step runnable in the field - prepare: add log-storage-operator, and cap Kafka disk with retention.bytes / log.retention.bytes - install: add the Log Essentials and Log Collector cluster plugins - add the Chinese translations --- docs/en/install/index.mdx | 314 +++++-- docs/en/prepare/index.mdx | 1683 ++++++++++++++++++++---------------- docs/zh/install/index.mdx | 323 +++++-- docs/zh/prepare/index.mdx | 1685 +++++++++++++++++++++---------------- 4 files changed, 2432 insertions(+), 1573 deletions(-) diff --git a/docs/en/install/index.mdx b/docs/en/install/index.mdx index 51a2610..e5f6e37 100644 --- a/docs/en/install/index.mdx +++ b/docs/en/install/index.mdx @@ -4,97 +4,140 @@ weight: 14 # Installation -This chapter installs the logging components against the storage and message queue that you operate: a ClickHouse or OpenSearch cluster and a Kafka service. Prepare them first, including their accounts, ACLs, and topics: +This chapter installs the logging components against the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service from [Environment Preparation](../prepare/index.mdx). -- [Environment Preparation](../prepare/index.mdx) +Step 1 is required for ClickHouse and OpenSearch. Then run either Step 2 (ClickHouse) or Step 3 (OpenSearch): the two are mutually exclusive, and each creates the connection Secret and the `PlatformLogForward`. Step 4 is required for both storage types and deploys the two cluster plugins after `PlatformLogForward` is ready. -The connection details you recorded there are the inputs to the Secrets below. - -Use this path when the log, event, and audit data must live in storage you operate — a ClickHouse or an OpenSearch cluster, plus a Kafka service — instead of the storage that the platform storage plugins install. OpenSearch as a log storage target is only available through this path. - -## Prepare the storage first - -Create the target cluster and the Kafka service, including their accounts, ACLs, and topics, and record the connection details: - -- [Environment Preparation](../prepare/index.mdx) - -Those recorded values are the inputs to the Secrets below. - -## Step 1: Create the connection Secrets +## Step 1: Create the Kafka connection Secret **Run on the workload cluster.** -Create the Secret for your target storage and the Secret for the Kafka service in `cpaas-system`. Create only the storage Secret that matches your target. - -### Target OpenSearch - ```yaml apiVersion: v1 kind: Secret metadata: - name: platform-default-os-conn # Connection Secret name, referenced when you create the PlatformLogForward later - namespace: cpaas-system -type: Opaque -stringData: - endpoints: "https://:9200" # Required; comma-separated HTTP(S) URLs; put the highly available coordinator or load balancer first - username: "" # Can be omitted when the target allows anonymous access - password: "" # Can be omitted when the target allows anonymous access ---- -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-mq-conn # Kafka connection Secret name, referenced when you create the PlatformLogForward later + name: platform-default-mq-conn namespace: cpaas-system type: Opaque stringData: - bootstrap: "" # Required; Kafka addresses in host:port form, separated by commas - kafkaClusterName: "" # Required; Kafka broker resource name; must match the actual name - username: "" # Required; Kafka user name - password: "" # Required; at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts - sasl_mechanism: "SCRAM-SHA-512" # Optional, defaults to SCRAM-SHA-512 - topics.log: "ALAUDA_LOG_TOPIC" # Optional, log topic name, defaults to ALAUDA_LOG_TOPIC - topics.event: "ALAUDA_EVENT_TOPIC" # Optional, event topic name, defaults to ALAUDA_EVENT_TOPIC - topics.audit: "ALAUDA_AUDIT_TOPIC" # Optional, audit topic name, defaults to ALAUDA_AUDIT_TOPIC - tls.ca: |- # Required when Kafka uses TLS and its certificate is not trusted by the system + # Required. Kafka addresses in host:port form, comma-separated for multiple brokers. + # Read from the service created in step 3 of Environment Preparation. + # Default: cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093 (SASL over TLS) + # or cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9092 (SASL without TLS) + bootstrap: "cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093" + # Required. metadata.name of the RdsKafka resource from Environment Preparation. + kafkaClusterName: "cpaas-kafka" + # Required. Name of the RdsKafkaUser from Environment Preparation. + username: "platform-logging" + # Required. The password you set in secret/platform-logging-password, key password. + # Must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts. + password: "" + # Optional. Defaults to SCRAM-SHA-512. Allowed: SCRAM-SHA-512, SCRAM-SHA-256, PLAIN. + sasl_mechanism: "SCRAM-SHA-512" + # Optional. Topic names, defaults shown. They must match the topics created in + # step 3 of Environment Preparation. + topics.log: "ALAUDA_LOG_TOPIC" + topics.event: "ALAUDA_EVENT_TOPIC" + topics.audit: "ALAUDA_AUDIT_TOPIC" + # Optional. Only for a TLS listener whose certificate the platform does not trust. + # Read the PEM from secret/-cluster-ca-cert, key ca.crt. + # Omit this whole field for the plain listener (:9092) or a system-trusted CA. + tls.ca: |- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ``` -### Target ClickHouse +Read the values you do not have at hand: + +```bash +# Kafka password +kubectl -n cpaas-system get secret platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d + +# Kafka cluster service (bootstrap address) +kubectl -n cpaas-system get svc cpaas-kafka-kafka-bootstrap + +# Kafka CA, only when the listener uses TLS and the CA is not system-trusted +kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert \ + -o jsonpath='{.data.ca\.crt}' | base64 -d +``` + +Save the YAML as `platform-default-mq-conn.yaml` and apply it: + +```bash +kubectl apply -f platform-default-mq-conn.yaml +``` + +Do not set `tls.insecure_skip_verify: "true"`; on the plain listener omit `tls.ca` instead. + +## Step 2: ClickHouse: Install the Logging Components + +This step applies to ClickHouse only. For OpenSearch, use Step 3 instead. + +### 2.1 Create the ClickHouse connection Secret + +**Run on the workload cluster.** ```yaml apiVersion: v1 kind: Secret metadata: - name: platform-default-ch-conn # Connection Secret name, referenced when you create the PlatformLogForward later + name: platform-default-ch-conn namespace: cpaas-system type: Opaque stringData: - endpoint: "https://:8443" # Required; ClickHouse address, including protocol and port - cluster: "replicated" # Must match the ClickHouse cluster name; ACP baseline default is replicated - database: "observability" # Target database name, defaults to observability - username: "" # ClickHouse user name - password: "" # ClickHouse password - tls.ca: |- # Required when the target uses HTTPS with a private CA + # Required. ClickHouse HTTP address including scheme and port: 8123 for HTTP, + # 8443 for HTTPS. Read it from the cluster Service created in step 4 of + # Environment Preparation: + # kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + endpoint: "http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123" + # Must match spec.configuration.clusters[0].name of the ClickHouseInstallation + # and the cluster name used by ON CLUSTER. ACP baseline default: replicated. + cluster: "replicated" + # ClickHouse database. Default: observability (from the ClickHouseInstallation). + database: "observability" + # The logging account declared in the ClickHouseInstallation. + # Default: platform-logging. + username: "platform-logging" + # Password of that account, stored in secret/clickhouse-platform-logging-password. + password: "" + # Optional. Only when the endpoint uses HTTPS with a private CA. + # Omit this whole field for the plain HTTP endpoint (:8123). + tls.ca: |- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ``` -`endpoints` accepts multiple comma-separated HTTP(S) URLs. Some data paths use only the first address, so put the highly available load balancer or coordinator endpoint first, not a single data node, and do not add leading whitespace before the first URL. +Read the values you do not have at hand: -The OpenSearch Secret carries no TLS key. The platform connects to OpenSearch with certificate verification disabled, so an endpoint behind a private CA needs no entry here; use `https://` in `endpoints` and the connection is established regardless of the issuer. +```bash +# ClickHouse service and port +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse -For ClickHouse, do not set `tls.insecure_skip_verify: "true"`; provide `tls.ca` instead so the platform can verify the target. +# Cluster name, database, and shard / replica counts +kubectl -n cpaas-system get chi cpaas-clickhouse \ + -o jsonpath='{.spec.configuration.clusters[0].name}{"\t"}{.spec.configuration.settings.default_database}{"\t"}{.spec.configuration.clusters[0].layout.shardsCount}{"\t"}{.spec.configuration.clusters[0].layout.replicasCount}{"\n"}' -## Step 2: Create the PlatformLogForward +# Logging account password +kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d +``` -**Run on the workload cluster.** +Save the YAML as `platform-default-ch-conn.yaml` and apply it: + +```bash +kubectl apply -f platform-default-ch-conn.yaml +``` + +Do not set `tls.insecure_skip_verify: "true"`. When the endpoint uses HTTPS, provide `tls.ca` so the platform can verify the server certificate; when it uses plain HTTP, omit `tls.ca`. -Create one `PlatformLogForward` for your target. +### 2.2 Create the PlatformLogForward + +**Run on the workload cluster.** -### Target OpenSearch +The `output.type` field is required for ClickHouse. ```yaml apiVersion: log.alauda.io/v1alpha1 @@ -103,21 +146,109 @@ metadata: name: platform-default # Fixed cluster singleton name, do not change spec: installMode: Fresh # Always Fresh, do not change it to Adopt + output: + type: clickhouse # Required for ClickHouse externalStorage: - type: opensearch # Target storage type + type: clickhouse # Storage type + shards: 1 # Set from the profile table below: 1 for a single node, 2 for six nodes, 3 for nine nodes + replicas: 1 # Set from the profile table below: 1 for a single node, 3 for three nodes and above secretRef: - name: platform-default-os-conn # Target storage connection Secret created in Step 1 + name: platform-default-ch-conn # Connection Secret created in Step 2.1 namespace: cpaas-system externalMessageQueue: type: kafka # Message queue type, currently only kafka secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + name: platform-default-mq-conn # Kafka connection Secret created in Step 1 namespace: cpaas-system ``` -### Target ClickHouse +`externalStorage.shards` and `externalStorage.replicas` must match the ClickHouse profile you deployed in Environment Preparation: + +| Profile | `shards` | `replicas` | +| --- | --- | --- | +| Single node (evaluation only) | 1 | 1 | +| Three nodes | 1 | 3 | +| Six nodes | 2 | 3 | +| Nine nodes | 3 | 3 | + +A wrong value in a multi-shard or replicated deployment leaves part of the ClickHouse topology unused. -The `output.type` field is required for this target. +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. These are the replica counts of the logging components themselves, not of ClickHouse: a single-node ClickHouse still runs them as they are. + +Save the YAML as `platform-log-forward.yaml` and apply it: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +### 2.3 Verify + +Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column reaches `Ready` and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +`PlatformLogForward` is ready. Continue to Step 4 to deploy the cluster plugins, then run the end-to-end data check in Step 4.3. + +## Step 3: OpenSearch: Install the Logging Components + +This step applies to OpenSearch only. For ClickHouse, use Step 2 instead. + +### 3.1 Create the OpenSearch connection Secret + +**Run on the workload cluster.** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn + namespace: cpaas-system +type: Opaque +stringData: + # Required. Comma-separated HTTP(S) URLs. Put the highly available + # coordinator or load balancer first: some data paths use only the first one. + # Read it from the OpenSearch service created in step 5 of Environment + # Preparation, for example https://cpaas-opensearch.cpaas-system.svc:9200 + # or the load balancer in front of it. Do not add leading whitespace. + endpoints: "https://:9200" + # The account created in step 5 of Environment Preparation. + # Default: platform-logging. Omit both fields when the cluster allows + # anonymous access. + username: "" + password: "" +``` + +Read the values you do not have at hand: + +```bash +# OpenSearch service and port +kubectl -n cpaas-system get svc cpaas-opensearch + +# Cluster health +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' +``` + +Save the YAML as `platform-default-os-conn.yaml` and apply it: + +```bash +kubectl apply -f platform-default-os-conn.yaml +``` + +This Secret has no `tls.ca` key: the platform connects to OpenSearch with certificate verification disabled, so an endpoint behind a private CA needs no entry here. Use `https://` in `endpoints` and the connection is established regardless of the issuer. + +### 3.2 Create the PlatformLogForward + +**Run on the workload cluster.** ```yaml apiVersion: log.alauda.io/v1alpha1 @@ -126,24 +257,18 @@ metadata: name: platform-default # Fixed cluster singleton name, do not change spec: installMode: Fresh # Always Fresh, do not change it to Adopt - output: - type: clickhouse # Required when the target is ClickHouse externalStorage: - type: clickhouse # Target storage type - shards: 1 # Actual shard count of the target ClickHouse - replicas: 1 # Actual replica count of the target ClickHouse + type: opensearch # Storage type secretRef: - name: platform-default-ch-conn # Target storage connection Secret created in Step 1 + name: platform-default-os-conn # Connection Secret created in Step 3.1 namespace: cpaas-system externalMessageQueue: type: kafka # Message queue type, currently only kafka secretRef: - name: platform-default-mq-conn # Kafka connection Secret created in Step 1, topic names come from this Secret + name: platform-default-mq-conn # Kafka connection Secret created in Step 1 namespace: cpaas-system ``` -`externalStorage.shards` and `externalStorage.replicas` must match the actual ClickHouse topology. Both default to `1`; a wrong value in a multi-shard or replicated deployment leaves part of the target topology unused. - `PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. Save the YAML as `platform-log-forward.yaml` and apply it: @@ -152,7 +277,7 @@ Save the YAML as `platform-log-forward.yaml` and apply it: kubectl apply -f platform-log-forward.yaml ``` -## Step 3: Verify +### 3.3 Verify Watch the status until it finishes, and press `Ctrl+C` to stop: @@ -167,4 +292,51 @@ kubectl get platformlogforward platform-default \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` -Then produce or locate new log, event, and audit records and confirm that you can query them from the target storage. +`PlatformLogForward` is ready. Continue to Step 4 to deploy the cluster plugins, then run the end-to-end data check in Step 4.3. + +## Step 4: Deploy the Cluster Plugins + +**Run after `PlatformLogForward/platform-default` is `Ready=True`.** + +This step is required for both ClickHouse and OpenSearch. `PlatformLogForward` installs the storage and forwarding data path; it does not install these two cluster plugins. + +### 4.1 Deploy Log Essentials + +**Run on the global cluster.** + +1. Open **Marketplace** > **Cluster Plugins** and select `global`. +2. Install **Alauda Container Platform Log Essentials** with the default configuration. +3. Verify that the plugin is running: + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=log-api + ``` + +The `STATUS` column must be `Running`. + +### 4.2 Deploy Log Collector + +Run this step on every cluster whose logs, events, and audit records must be collected. If the global cluster's own data must be collected, install it on `global` as well. + +1. Open **Marketplace** > **Cluster Plugins** and select the cluster. +2. Install **Alauda Container Platform Log Collector** and set: + +| Field | Value | +| --- | --- | +| **Log Center Plugin** | **Standard**. This is the data path managed by `log-storage-operator`; it applies to both ClickHouse and OpenSearch. Do not select the legacy `ClickHouse` or `ElasticSearch` option. | +| **Storage Cluster Name** | Enter the name of the cluster where `PlatformLogForward/platform-default` is `Ready`. When the collector and storage are in the same cluster, enter that cluster's name. This field is a text input and is not auto-discovered for the `log-storage-operator` path. | +| **Log Collector Storage Path** | Enter an absolute path for the collector's local working data. Use `/cpaas` on a traditional OS. On Alauda OS, use a writable path under `/var/cpaas`, for example `/var/cpaas`. | +| **Mount Paths** | Optional. Add absolute host paths that contain log files the collector must read. | +| **Audit**, **Event**, **Kubernetes**, **Platform**, **System**, **Workload** | Select the log types to collect. The defaults enable audit, event, system, and workload collection; Kubernetes and platform collection are disabled. | + +3. Install the plugin and verify that it is running: + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=logagent + ``` + +Run the verification command from the global cluster. The row for each collector cluster must show `STATUS=Running`. + +### 4.3 Verify the Data Path + +After both plugin checks pass, produce or locate new log, event, and audit records and confirm that they can be queried from ClickHouse or OpenSearch. If no records arrive, inspect the `Log Collector` `ModuleInfo` status and the `PlatformLogForward` conditions before changing either resource. diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 4d59a34..8e993ef 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -4,7 +4,7 @@ weight: 13 # Environment Preparation -This chapter explains how to set up the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service that the logging components use: nodes and disks, the operators, the cluster, its accounts, and the Kafka password, user, ACLs, and topics. +This chapter prepares the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service that the logging components use. Steps 1 to 3 apply to ClickHouse and OpenSearch alike. Step 4 applies only to ClickHouse and Step 5 only to OpenSearch: run only the one you use. Follow the steps in order and complete each verification. @@ -14,15 +14,15 @@ Make sure you have: 1. Administrator access to the cluster that runs the logging components. 2. The nodes and disks planned in Step 1. -3. The operator packages available in the platform marketplace: `clickhouse-operator`, the Alauda Kafka operator, and `opensearch-operator`. +3. The operator packages available in the platform marketplace: `log-storage-operator`, `clickhouse-operator`, the Alauda Kafka operator, and `opensearch-operator`. Use [Log Component Capacity Planning](../architecture/capacity_planning.mdx) with the tables below to choose the scale, and place the workloads on dedicated nodes as described in [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). -Run every command in this chapter from a host that has `kubectl` access to the cluster. Each YAML block is a file you save and then apply; the text below each block gives the file name and the `kubectl apply -f` command. +Run every command in this chapter from a host that has `kubectl` access to the cluster. Each YAML block is a file you save and then apply; each step gives the file name and the `kubectl apply -f` command. -## Step 0: Choose the Target and the Scale +## Step 0: Choose ClickHouse or OpenSearch and the Scale -Choose the target (ClickHouse or OpenSearch) and the scale. +Choose ClickHouse or OpenSearch, and the scale. Both need the Kafka service in Step 3, and Steps 4 and 5 are mutually exclusive. ### ClickHouse profiles @@ -35,11 +35,11 @@ The CPU and memory values are the container limits for each ClickHouse pod. | Six nodes | 6 | 2 shards × 3 replicas | 4C | 8G | 40,000 logs/s | | Nine nodes | 9 | 3 shards × 3 replicas | 4C | 8G | 69,000 logs/s | -Use the single-node profile for evaluation only. Start production at the three-node profile, and move to six or nine nodes when a single shard no longer fits. +Use the single-node profile for evaluation only, with the single-node Kafka manifest in Step 3. Start production at the three-node profile, and move to six or nine nodes when a single shard no longer fits. ### Kafka -Set up three brokers with a 2C/4G limit each, plus the three controllers that the manifest in Step 4 runs at 1C/2G. Size the broker volumes by retention and throughput. +Set up three brokers with a 2C/4G limit each, plus the three controllers that the manifest in Step 3 runs at 1C/2G. Size the broker volumes by retention and throughput. The single-node evaluation profile runs one broker and one controller with the same limits. ### OpenSearch profiles @@ -52,7 +52,7 @@ The CPU and memory values are per-node limits. | Large scale | 3 + 5 | 3 master, 5 data | 2C master / 8C data | 4G master / 16G data | 25,000 logs/s | | Large scale | 3 + 7 | 3 master, 7 data | 2C master / 8C data | 4G master / 16G data | 30,000 logs/s | -Do not size below the smallest profile, and use the large-scale profiles once a single node pool can no longer serve the data volume. If your measured storage is weaker than 6,000 IOPS and 250 MB/s read/write, size up. +The smallest supported OpenSearch profile is three nodes; do not deploy OpenSearch as a single node. Do not size below the smallest profile, and use the large-scale profiles once a single node pool can no longer serve the data volume. If your measured storage is weaker than 6,000 IOPS and 250 MB/s read/write, size up. ### Disk @@ -66,52 +66,22 @@ Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, - On the traditional operating system layout, use `/cpaas/data/...`. - On Alauda OS nodes only `/var/cpaas` is writable, so use `/var/cpaas/data/...`. 4. Make sure the path survives node re-provisioning. -5. OpenSearch needs `vm.max_map_count` set to at least `262144`. The OpenSearch operator sets it from an init container, so nothing is required here unless your cluster enforces restricted Pod Security Admission, in which case that init container cannot run and you must set it on every node yourself: - - ```bash - sudo sysctl -w vm.max_map_count=262144 - echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf - ``` -6. Create the directories the storage pods use and set their ownership. The examples use the traditional layout; on Alauda OS nodes replace `/cpaas` with `/var/cpaas`. - - ```bash - # ClickHouse runs as uid 101 - sudo mkdir -p /cpaas/data/clickhouse - sudo chown -R 101:101 /cpaas/data/clickhouse - - # Kafka runs as uid 1001; one directory per pod, see below - sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-3 - sudo chown -R 1001:1001 /cpaas/data/kafka - - # OpenSearch runs as uid 1000 - sudo mkdir -p /cpaas/data/opensearch - sudo chown -R 1000:1000 /cpaas/data/opensearch - ``` - -7. Decide how the volumes are provisioned: +5. Decide how the volumes are provisioned: | Approach | When to use it | What you must do | | --- | --- | --- | -| Static local volumes | You are pinning each pod to a specific node, which is what the infra-node setup usually does | Create one StorageClass without a provisioner, and pre-create one PV per intended pod, each with `nodeAffinity` and `local.path` pointing at the directory above | +| Static local volumes | You are pinning each pod to a specific node, which is what the infra-node setup usually does | Create one StorageClass without a provisioner, and pre-create one PV per intended pod, each with `nodeAffinity` and `local.path` pointing at that pod's directory | | Dynamic provisioner | Your platform provides a block-storage provisioner | Create the StorageClass and let the claims bind dynamically; confirm the provisioner supports `ReadWriteOnce` block volumes and the throughput above | For static local volumes, create one StorageClass and one PV per pod, and **reserve every PV for the claim it belongs to**. A `local` volume cannot follow its pod: unless a volume is reserved, a claim can bind a volume that was prepared for another pod or component, and after an instance is deleted and recreated its pods can bind each other's disks. Reserving the volume with `spec.claimRef` removes that risk, because the volume then only ever matches the claim named in it. -Use a separate StorageClass per component. The example below creates the one for ClickHouse and reserves a single volume for the first ClickHouse replica. The number of PVs is the sum of the pods you plan: ClickHouse `shardsCount × replicasCount`, Kafka `replicas + controller.replicas`, and OpenSearch the sum of the node pool `replicas`. - -Claim names are deterministic for ClickHouse and OpenSearch, so their volumes can be reserved before the instance exists: - -| Component | Claim name for replica `` | -| --- | --- | -| ClickHouse | `data-volumeclaim-template-chi-----0` | -| OpenSearch | `data---` | -| Kafka | `data--broker--`, so read the names from the instance first — see Step 4 | +Use a separate StorageClass per component, so one component's claims cannot bind a volume that was prepared for another. The number of PVs is the sum of the pods you plan: ClickHouse `shardsCount × replicasCount`, Kafka `replicas + controller.replicas`, and OpenSearch the sum of the node pool `replicas`. The shape of the manifest is the same for every component, and each component's step gives the exact claim name, directory, and size: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: - name: cpaas-local-clickhouse + name: cpaas-local- labels: # Required in a project namespace: without the grant, the pvc-validator # admission webhook rejects every claim using this class. @@ -124,22 +94,22 @@ allowVolumeExpansion: false apiVersion: v1 kind: PersistentVolume metadata: - name: cpaas-clickhouse-0 + name: spec: capacity: - storage: 200Gi + storage: volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain - storageClassName: cpaas-local-clickhouse + storageClassName: cpaas-local- claimRef: # Reserve this volume for exactly this claim apiVersion: v1 kind: PersistentVolumeClaim namespace: cpaas-system - name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + name: local: - path: /cpaas/data/clickhouse + path: nodeAffinity: required: nodeSelectorTerms: @@ -149,314 +119,770 @@ spec: values: [""] ``` -Save the YAML as `local-storage.yaml` and apply it: - -```bash -kubectl apply -f local-storage.yaml -``` - -Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path` and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example `/cpaas/data/clickhouse-0` and `/cpaas/data/clickhouse-1`, created with the ownership from step 6. +Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path` and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory. `capacity.storage` on a `local` volume is matching metadata, not a quota: nothing stops a pod from filling the underlying disk past it. Set it to the real usable size and enforce retention on the storage side as well. ## Step 2: Install the Operators -Install the three operators from the platform marketplace. Every storage and messaging resource below is created in `cpaas-system`, so each operator must be able to reconcile resources in that namespace. +Install the operators from the platform marketplace. The `ClickHouseInstallation`, `RdsKafka`, and OpenSearch resources below and the `PlatformLogForward` used during installation all live in `cpaas-system`, so every operator must be able to reconcile resources in that namespace. Install the ClickHouse operator only for ClickHouse and the OpenSearch operator only for OpenSearch; every deployment also needs the Kafka operator and `log-storage-operator`. | Operator | Subscription namespace | Namespaces the operator must watch | | --- | --- | --- | -| `clickhouse-operator` | `cpaas-system` | `cpaas-system` | +| `log-storage-operator` | `cpaas-system` | `cpaas-system` | +| `clickhouse-operator` (ClickHouse only) | `cpaas-system` | `cpaas-system` | | Alauda Kafka operator (`strimzi-kafka-operator`) | `kafka-system` | All namespaces | -| `opensearch-operator` (OpenSearch target only) | `opensearch-operator` | All namespaces | +| `opensearch-operator` (OpenSearch only) | `opensearch-operator` | All namespaces | - Install only the operators that are missing. If one is already installed on the cluster, for example by an earlier release, keep it and do not install a second copy: two copies of the same operator write to the same `cpaas-system` resources. Check its watch scope instead and widen it if needed. - Do not create an OperatorGroup in `cpaas-system`. The platform already owns one there, and a second OperatorGroup makes the platform reject every Subscription in that namespace, including its own. - For `kafka-system` and `opensearch-operator`, the OperatorGroup must have no `spec.targetNamespaces`. If an OperatorGroup scoped to its own namespace already exists, remove the field and wait for the operator pod to restart. - ```bash - kubectl -n kafka-system patch operatorgroup kafka-system \ - --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' - kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ - --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' - ``` +:::warning +An operator that does not watch `cpaas-system` ignores the resources below silently: no status, no events, and no pods. Confirm the deployments are ready and that the Kafka and OpenSearch OperatorGroups reach all namespaces before you continue. +::: + +### log-storage-operator + +Required for both. It provides the `PlatformLogForward` that Installation creates. + +```bash +kubectl get crd platformlogforwards.log.alauda.io logforwards.log.alauda.io +kubectl -n cpaas-system get deploy log-storage-operator-controller-manager +kubectl -n cpaas-system get sub log-storage-operator +``` -Verify each operator before you continue: +### ClickHouse operator + +Only for ClickHouse. The OperatorGroup already exists in `cpaas-system`, so there is nothing to patch. ```bash -# ClickHouse kubectl get crd clickhouseinstallations.clickhouse.altinity.com kubectl -n cpaas-system get deploy clickhouse-operator +``` + +### Kafka operator + +Required for both. + +```bash +kubectl -n kafka-system patch operatorgroup kafka-system \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' -# Kafka kubectl get crd rdskafkas.middleware.alauda.io kubectl -n kafka-system get deploy strimzi-cluster-operator +``` + +### OpenSearch operator + +Only for OpenSearch. + +```bash +kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' -# OpenSearch (only when the target is OpenSearch) kubectl get crd opensearchclusters.opensearch.opster.io kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager ``` -:::warning -An operator that does not watch `cpaas-system` ignores the resources below silently: no status, no events, and no pods. Confirm the deployments are ready and that the Kafka and OpenSearch OperatorGroups reach all namespaces before you continue. -::: - -## Step 3: Create the ClickHouse Cluster - -Skip this step when the target is OpenSearch. +## Step 3: Create the Kafka Service -### 3.1 Create the password Secrets +### 3.1 Create the SASL password Secret -The instance defines two accounts: `admin` for administration, and `platform-logging` for the logging components. Both read their password from a Secret, so create both before you create the instance. +The password must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts. ```bash -kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ - --from-literal=password="$(openssl rand -hex 16)" -kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ +kubectl -n cpaas-system create secret generic platform-logging-password \ --from-literal=password="$(openssl rand -hex 16)" ``` -### 3.2 Create the Keeper client Service (three nodes and above) +### 3.2 Create the broker cluster -Every profile needs a Keeper, including the single-node profile, because the logging components create `ReplicatedMergeTree` tables. The single-node profile runs the Keeper inside its ClickHouse pod through the `keeper_server/*` settings in 3.3, so skip this step for it. +Apply the manifest for the profile you chose in Step 0: three nodes and above is the production profile, single node is for evaluation only. The two differ only in the replica counts and the replication factors; do not mix them. -For three nodes and above, the ClickHouse pods themselves form the Keeper quorum: every ClickHouse pod is also a Keeper member. ClickHouse reaches that quorum through a headless Service that selects all ready pods of the installation. +Do not omit these settings: -Save the YAML as `cpaas-clickhouse-keeper-service.yaml` and apply it: +| Setting | Why it is required | +| --- | --- | +| `message.max.bytes: "10485760"` | Audit batches are about 1.1–1.5 MiB each. The Kafka default of 1 MiB rejects every audit batch. | +| `replica.fetch.max.bytes: "10485760"` | Must be at least `message.max.bytes`, otherwise replica synchronization stalls. | +| `auto.create.topics.enable: "false"` | A mistyped topic name must not be created automatically and silently collect data. | +| `entityOperator.topicOperator` / `userOperator` | Without them, the `RdsTopic` and `RdsKafkaUser` resources in the next steps are not applied to the brokers. | + +#### 3.2.1 Three nodes and above (production) + +Create one directory per broker and per controller on the node that pod will run on, and set the ownership (uid 1001). On Alauda OS nodes replace `/cpaas` with `/var/cpaas`: + +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/broker-1 /cpaas/data/kafka/broker-2 +sudo mkdir -p /cpaas/data/kafka/controller-0 /cpaas/data/kafka/controller-1 /cpaas/data/kafka/controller-2 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` ```yaml -apiVersion: v1 -kind: Service +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka metadata: - name: cpaas-clickhouse-keeper + name: cpaas-kafka namespace: cpaas-system spec: - clusterIP: None - type: ClusterIP - ports: - - name: keeper - port: 9181 - protocol: TCP - targetPort: 9181 - selector: - clickhouse.altinity.com/chi: cpaas-clickhouse - clickhouse.altinity.com/namespace: cpaas-system - clickhouse.altinity.com/ready: "yes" - clickhouse.altinity.com/role: keeper + mode: KRaft + version: 4.2.0 # Minimum supported by the Alauda Kafka operator + replicas: 3 + resources: + limits: { cpu: "2", memory: 4Gi } # From the profile + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + controller: + replicas: 3 + roles: ["controller"] # Required: without it the node pool is rejected + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # Required: it creates the topics in Step 3.5 + userOperator: {} # Required: it creates the SASL user in Step 3.4 ``` +Save the YAML as `cpaas-kafka.yaml` and apply it: + ```bash -kubectl apply -f cpaas-clickhouse-keeper-service.yaml +kubectl apply -f cpaas-kafka.yaml ``` -The `chi`, `namespace` and `ready` labels are set by the operator. The `role: keeper` label comes from the pod template in 3.3. - -### 3.3 Create the ClickHouseInstallation - -Keep the cluster name `replicated` for the logging components, and set `shardsCount` and `replicasCount` from the Step 0 profile. Apply one of the two manifests below, depending on the profile you chose. +#### 3.2.2 Single node (evaluation only) -Replace `` with the ClickHouse server image published with the platform middleware packages, for example `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`. +Create one directory for the single broker and one for the single controller: -**Single node.** The Keeper runs inside the ClickHouse pod through the `keeper_server/*` settings. +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-0 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` ```yaml -apiVersion: clickhouse.altinity.com/v1 -kind: ClickHouseInstallation +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka metadata: - name: cpaas-clickhouse + name: cpaas-kafka namespace: cpaas-system spec: - configuration: - users: - # The admin password comes from the Secret created above. - admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password - admin/networks/ip: - - "0.0.0.0/0" - - "::/0" - admin/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION - # The account the logging components use. It is declared here, like the admin - # account, so it carries the privileges the components need without a separate - # set of GRANT statements. - platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password - platform-logging/networks/ip: - - "0.0.0.0/0" - - "::/0" - platform-logging/profile: default - platform-logging/quota: default - platform-logging/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION + mode: KRaft + version: 4.2.0 # Minimum supported by the Alauda Kafka operator + replicas: 1 + resources: + limits: { cpu: "2", memory: 4Gi } # From the profile + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + controller: + replicas: 1 + roles: ["controller"] # Required: without it the node pool is rejected + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "1" + min.insync.replicas: "1" + offsets.topic.replication.factor: "1" + transaction.state.log.replication.factor: "1" + transaction.state.log.min.isr: "1" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # Required: it creates the topics in Step 3.5 + userOperator: {} # Required: it creates the SASL user in Step 3.4 +``` - profiles: - default/allow_nondeterministic_mutations: "1" - default/allow_unrestricted_reads_from_keeper: "1" - default/max_execution_time: 120 - default/max_estimated_execution_time: 120 +Save the YAML as `cpaas-kafka.yaml` and apply it: - clusters: - - name: replicated # Reused in the connection Secret - templates: - podTemplate: pod-template - dataVolumeClaimTemplate: data-volumeclaim-template - layout: - shardsCount: 1 # From the profile: 1, 1, 2, or 3 - replicasCount: 1 # The example runs one pod; use 3 for three nodes and above +```bash +kubectl apply -f cpaas-kafka.yaml +``` - settings: - default_database: observability # Reused in the connection Secret - merge_tree/materialize_ttl_recalculate_only: "1" - # Self-observability system tables grow without bound and eventually fill the volume. - asynchronous_metric_log/database: system - asynchronous_metric_log/table: asynchronous_metric_log - asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" - metric_log/database: system - metric_log/table: metric_log - metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" - trace_log/database: system - trace_log/table: trace_log - trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" - # Co-located Keeper for the single-node profile. For three nodes and above, - # use the manifest below instead, which runs Keeper in every ClickHouse pod. - keeper_server/tcp_port: "9181" - keeper_server/server_id: "1" - keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log - keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots - keeper_server/coordination_settings/operation_timeout_ms: "10000" - keeper_server/coordination_settings/session_timeout_ms: "30000" - keeper_server/raft_configuration/server/id: "1" - keeper_server/raft_configuration/server/hostname: localhost - keeper_server/raft_configuration/server/port: "9234" +#### 3.2.3 Reserve the volumes for the broker claims (both profiles) - zookeeper: - nodes: - - host: localhost - port: 9181 +The instance creates its claims immediately. They stay `Pending` until you reserve volumes for them, because the broker claim names contain a hash that is generated per instance, which is why the volumes cannot be prepared up front: - defaults: - templates: - podTemplate: pod-template - dataVolumeClaimTemplate: data-volumeclaim-template - serviceTemplate: service-template +```bash +kubectl -n cpaas-system get pvc \ + -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' +``` - templates: - podTemplates: - - name: pod-template - podDistribution: - - scope: Shard - topologyKey: kubernetes.io/hostname - type: ShardAntiAffinity - spec: - nodeSelector: - node-role.kubernetes.io/infra: "" # The label you set in Step 1 - tolerations: - - key: node-role.kubernetes.io/infra # The taint you set in Step 1 - operator: Exists - effect: NoSchedule - containers: - - name: clickhouse - image: - ports: - - name: http - containerPort: 8123 - - name: client - containerPort: 9000 - - name: interserver - containerPort: 9009 - - name: keeper - containerPort: 9181 - - name: raft - containerPort: 9234 - resources: - requests: - cpu: "1" - memory: 4Gi - limits: - cpu: "2" # From the profile - memory: 4Gi # From the profile - volumeMounts: - - name: data-volumeclaim-template - mountPath: /var/lib/clickhouse +Copy every claim name — three brokers and three controllers for the three-node profile, one broker and one controller for the single-node profile — and create one pre-bound volume per claim. Set `capacity.storage` to the size the claim asks for, point `local.path` at that pod's directory, and pin the node: - serviceTemplates: - - name: service-template - spec: - ports: - - name: http - port: 8123 - - name: tcp - port: 9000 - type: ClusterIP +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-kafka-broker-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-kafka + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-cpaas-kafka-broker--0 + local: + path: /cpaas/data/kafka/broker-0 + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` - volumeClaimTemplates: - - name: data-volumeclaim-template - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 200Gi - storageClassName: # From Step 1 +Save the YAML as `cpaas-kafka-volumes.yaml`, repeat the `PersistentVolume` part for every claim, and apply it. The claims bind as soon as their volumes exist, and the brokers and controllers start: + +```bash +kubectl apply -f cpaas-kafka-volumes.yaml ``` -Save the YAML as `cpaas-clickhouse.yaml` and apply it: +### 3.3 Wait for the broker cluster ```bash -kubectl apply -f cpaas-clickhouse.yaml +kubectl -n cpaas-system get rdsKafka cpaas-kafka \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' +kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka ``` -**Three nodes and above.** Use this manifest instead of the one above. The Keeper runs inside every ClickHouse pod, so the pods form the quorum among themselves and the installation stays a single `ClickHouseInstallation`. +Wait until the `Ready` condition is `True` and all broker pods are `Running`. -The static Keeper configuration is injected through the cluster `files` and pulls in a generated file with `include_from`. The identity-dependent part — `server_id` and the member list — is generated per pod by an init container into an in-memory `emptyDir`. Keep `SHARDS_COUNT` and `REPLICAS_COUNT` in that init container equal to `layout.shardsCount` and `layout.replicasCount`, otherwise the member list is incomplete and the quorum never forms. +Confirm the required settings reached the brokers: -The readiness probe checks the Raft port. This is required: the default HTTP probe only succeeds after ClickHouse is serving, and ClickHouse does not finish starting until the Keeper quorum exists, so the operator would wait for the first replica forever and never create the remaining ones. +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" -The Keeper `path` sits under `/var/lib/clickhouse`, which is the mounted data volume, so the Keeper log and snapshots live on the persistent volume together with the ClickHouse data. Do not move it outside that mount: Keeper state kept in the container filesystem is lost whenever the pod restarts. +kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ + grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes|^log\.retention\.bytes" /tmp/strimzi.properties +``` -The `wait-for-self-dns` init container waits until the pod resolves its own headless service name. Without it, a pod that starts before its DNS record is published initialises its distributed DDL worker against an unresolved hostname and then never retries: `CREATE TABLE ... ON CLUSTER` succeeds on the other replicas, and that replica silently misses the statement. +`message.max.bytes` and `replica.fetch.max.bytes` must be `10485760`, and `log.retention.bytes` must match the topic cap in 3.5, otherwise stop. Read the broker values from this file: `kafka-configs.sh --describe` does not show the two static settings. + +Then check the node placement for your profile. + +#### 3.3.1 Three nodes and above (production) + +The Kafka operator applies hard pod anti-affinity, so the three brokers must land on three different nodes: + +```bash +kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +``` + +Three distinct node names are required: hard anti-affinity needs at least three schedulable nodes. + +#### 3.3.2 Single node (evaluation only) + +The single broker and the single controller share one node, so one schedulable node is enough and there is no placement check to run. + +### 3.4 Create the SASL user and its ACLs + +The logging components use a single account with access to the three topics, the consumer groups, and the broker metadata. ```yaml -apiVersion: clickhouse.altinity.com/v1 -kind: ClickHouseInstallation +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser metadata: - name: cpaas-clickhouse + name: platform-logging namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka spec: - configuration: - users: - # The admin password comes from the Secret created above. - admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password - admin/networks/ip: - - "0.0.0.0/0" - - "::/0" - admin/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION - # The account the logging components use. It is declared here, like the admin - # account, so it carries the privileges the components need without a separate - # set of GRANT statements. - platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password - platform-logging/networks/ip: - - "0.0.0.0/0" - - "::/0" - platform-logging/profile: default - platform-logging/quota: default - platform-logging/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + # The three topics + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } + # The consumer groups used by the log pipeline + - host: "*" + operation: All + resource: { type: group, name: alauda_log, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_event, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_audit, patternType: literal } + # The consumer group prefix used by LogForward + - host: "*" + operation: All + resource: { type: group, name: "logforward-", patternType: prefix } + # The consumer group prefix used by the log query service + - host: "*" + operation: All + resource: { type: group, name: "razor-", patternType: prefix } + # Broker metadata + - host: "*" + operation: All + resource: { type: cluster, name: kafka-cluster, patternType: literal } +``` - profiles: - default/allow_nondeterministic_mutations: "1" - default/allow_unrestricted_reads_from_keeper: "1" - default/max_execution_time: 120 - default/max_estimated_execution_time: 120 +Save the YAML as `platform-logging-user.yaml` and apply it: - zookeeper: - nodes: - - host: cpaas-clickhouse-keeper # The Service created in 3.2 - port: 9181 +```bash +kubectl apply -f platform-logging-user.yaml +``` - settings: - default_database: observability # Reused in the connection Secret - merge_tree/materialize_ttl_recalculate_only: "1" - # Self-observability system tables grow without bound and eventually fill the volume. - asynchronous_metric_log/database: system +All nine entries are required. `operation: All` covers the read and describe permissions these entries need. Verify: + +```bash +kubectl -n cpaas-system get rdskafkauser platform-logging \ + -o jsonpath='{.status.phase}{"\n"}' # expect: Active +kubectl -n cpaas-system get secret platform-logging +``` + +The group names use underscores (`alauda_log`), while the topic names use uppercase letters and underscores (`ALAUDA_LOG_TOPIC`). + +### 3.5 Create the three topics + +The resource name must be a valid DNS name. `spec.topicName` is the broker-side name and must match the ACL entries above. + +`retention.ms` alone does not cap disk usage: one burst can fill the broker volume before the time window expires, and a full volume stops the log, event, and audit pipelines. Set `retention.bytes` as well. It is per partition, on both the topic and the broker, and is derived from the broker volume: + +``` +per-partition cap = volume size × 70% ÷ (number of topics × partitions per topic) +``` + +For the 200 Gi volume and three topics with 30 partitions each that is 200 Gi × 70% ÷ 90 = 1,670,265,059 bytes, rounded down to a whole 100 MiB segment: `1572864000`. The 30% reserve covers index files, `__consumer_offsets`, the `__cluster_metadata` replica, and the active segment that is never deleted. Use the same value for `retention.bytes` on all three topics and for `log.retention.bytes` on the brokers. Recompute it if you size the broker volume differently. + +#### 3.5.1 Three nodes and above (production) + +Create the three topics below as they are: `replicas: 3` and `min.insync.replicas: "2"`. + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC # Broker-side name; must match the ACL and the connection Secret + partitions: 30 # Upper bound for consumer parallelism + replicas: 3 + config: + retention.ms: "172800000" # 48 hours + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +``` + +Save the YAML as `alauda-topics.yaml` and apply it: + +```bash +kubectl apply -f alauda-topics.yaml +``` + +Confirm the caps were written to all three topics: + +```bash +kubectl -n cpaas-system get rdstopic \ + -o custom-columns='NAME:.metadata.name,TOPIC:.spec.topicName,PARTITIONS:.spec.partitions,RETENTION:.spec.config.retention\.bytes,SEGMENT:.spec.config.segment\.bytes' +``` + +`RETENTION` must be `1572864000` and `SEGMENT` `104857600` on every topic. `kafka-configs.sh --describe` omits the topic value when it equals the broker-level `log.retention.bytes`, so this is the reliable check. + +#### 3.5.2 Single node (evaluation only) + +Use the same three topics with `replicas: 1` and `min.insync.replicas: "1"`. + +### 3.6 Verify the Kafka service end to end + +Build the client properties inside the broker pod from the user Secret. The broker listeners require SASL, so every command below needs them. + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ + sh -c 'cat > /tmp/logging-client.properties' <----0`, and both `` and `` are zero-based. The manifest below is identical for the two profiles; only the number of PVs differs. With a dynamic provisioner, create only the StorageClass and skip the PVs. + +#### 4.1.1 Three nodes and above (production) + +The manifest in 4.4.1 is one shard and three replicas, so it needs three claims: `data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`, `-0-1-0` and `-0-2-0`. The manifest below reserves the first one: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local-clickhouse + labels: + # Required in a project namespace: without the grant, the pvc-validator + # admission webhook rejects every claim using this class. + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-clickhouse-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-clickhouse + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + local: + path: /cpaas/data/clickhouse + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +Save the YAML as `cpaas-clickhouse-volumes.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse-volumes.yaml +``` + +Repeat the `PersistentVolume` part for the other two replicas, with a distinct `metadata.name`, a distinct `local.path` (for example `/cpaas/data/clickhouse-1` and `/cpaas/data/clickhouse-2`) and the IP of the node that pod runs on. + +#### 4.1.2 Single node (evaluation only) + +The single-node profile has one pod, so it needs one claim: `data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`. The manifest above already reserves it; apply it as it is and do not add more PVs. + +### 4.2 Create the password Secrets + +The instance defines two accounts: `admin` for administration, and `platform-logging` for the logging components. Both read their password from a Secret, so create both before you create the instance. + +```bash +kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ + --from-literal=password="$(openssl rand -hex 16)" +kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 4.3 Create the Keeper client Service + +Every profile needs a Keeper, including the single-node profile, because the logging components create `ReplicatedMergeTree` tables. How ClickHouse reaches it differs by profile. + +#### 4.3.1 Three nodes and above (production) + +The ClickHouse pods themselves form the Keeper quorum: every ClickHouse pod is also a Keeper member, and ClickHouse reaches the quorum through a headless Service that selects all ready pods of the installation. + +Save the YAML as `cpaas-clickhouse-keeper-service.yaml` and apply it: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + clusterIP: None + type: ClusterIP + ports: + - name: keeper + port: 9181 + protocol: TCP + targetPort: 9181 + selector: + clickhouse.altinity.com/chi: cpaas-clickhouse + clickhouse.altinity.com/namespace: cpaas-system + clickhouse.altinity.com/ready: "yes" + clickhouse.altinity.com/role: keeper +``` + +```bash +kubectl apply -f cpaas-clickhouse-keeper-service.yaml +``` + +The `chi`, `namespace` and `ready` labels are set by the operator. The `role: keeper` label comes from the pod template in 4.4.1. + +#### 4.3.2 Single node (evaluation only) + +The Keeper runs inside the single ClickHouse pod through the `keeper_server/*` settings in 4.4.2, and ClickHouse reaches it on `localhost`. Do not create this Service; skip to 4.4. + +### 4.4 Create the ClickHouseInstallation + +Keep the cluster name `replicated` for the logging components, and set `shardsCount` and `replicasCount` from the Step 0 profile. Apply one of the two manifests below, depending on the profile you chose. + +Replace `` with the ClickHouse server image published with the platform middleware packages, for example `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`. + +#### 4.4.1 Three nodes and above (production) + +The Keeper runs inside every ClickHouse pod, so the pods form the quorum among themselves and the installation stays a single `ClickHouseInstallation`. + +The static Keeper configuration is injected through the cluster `files` and pulls in a generated file with `include_from`. The identity-dependent part — `server_id` and the member list — is generated per pod by an init container into an in-memory `emptyDir`. Keep `SHARDS_COUNT` and `REPLICAS_COUNT` in that init container equal to `layout.shardsCount` and `layout.replicasCount`, otherwise the member list is incomplete and the quorum never forms. + +The readiness probe checks the Raft port. This is required: the default HTTP probe only succeeds after ClickHouse is serving, and ClickHouse does not finish starting until the Keeper quorum exists, so the operator would wait for the first replica forever and never create the remaining ones. + +The Keeper `path` sits under `/var/lib/clickhouse`, which is the mounted data volume, so the Keeper log and snapshots live on the persistent volume together with the ClickHouse data. Do not move it outside that mount: Keeper state kept in the container filesystem is lost whenever the pod restarts. + +The `wait-for-self-dns` init container waits until the pod resolves its own headless service name. Without it, a pod that starts before its DNS record is published initialises its distributed DDL worker against an unresolved hostname and then never retries: `CREATE TABLE ... ON CLUSTER` succeeds on the other replicas, and that replica silently misses the statement. + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # The admin password comes from the Secret created above. + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # The account the logging components use. It is declared here, like the admin + # account, so it carries the privileges the components need without a separate + # set of GRANT statements. + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper # The Service created in 4.3.1 + port: 9181 + + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + # Self-observability system tables grow without bound and eventually fill the volume. + asynchronous_metric_log/database: system asynchronous_metric_log/table: asynchronous_metric_log asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" metric_log/database: system @@ -502,7 +928,7 @@ spec: type: ShardAntiAffinity metadata: labels: - clickhouse.altinity.com/role: keeper # Selected by the Service in 3.2 + clickhouse.altinity.com/role: keeper # Selected by the Service in 4.3.1 spec: nodeSelector: node-role.kubernetes.io/infra: "" # The label you set in Step 1 @@ -625,483 +1051,284 @@ spec: medium: Memory serviceTemplates: - name: service-template - spec: - ports: - - name: http - port: 8123 - - name: tcp - port: 9000 - type: ClusterIP - volumeClaimTemplates: - - name: data-volumeclaim-template - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 200Gi - storageClassName: # From Step 1 -``` - -Save the YAML as `cpaas-clickhouse.yaml` and apply it: - -```bash -kubectl apply -f cpaas-clickhouse.yaml -``` - -`default_database: observability` makes ClickHouse create the `observability` database at startup, so there is no database to create here. - -### 3.4 Wait for the cluster - -```bash -kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ - -o jsonpath='{.status.status}{"\n"}' # re-run until it reports: Completed - -kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse -kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse -kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse -``` - -All ClickHouse pods must be `Running` and ready, and every claim must be `Bound`. A StorageClass name that does not exist or cannot bind produces no pods and no error, so check the claims rather than the `ClickHouseInstallation` status alone. - -For three nodes and above, confirm the Keeper quorum before you continue. The Keeper runs inside the ClickHouse pods, so check one pod per replica: - -```bash -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -``` - -The three lines must report one `leader` and two `follower`. Also confirm that ClickHouse reads the quorum through the Service from 3.2: - -```bash -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ - clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" -# expect: cpaas-clickhouse-keeper 9181 -``` - -### 3.5 Read the logging account password - -The account is declared in the `ClickHouseInstallation` you applied in 3.3, so there is nothing to create here and no `GRANT` statements to run: like the `admin` account, it is a server-configuration user and carries the privileges the logging components need. - -Read its password for the connection details in Step 6. 3.6 reuses the `CH_POD` and `LOG_PASSWORD` variables, so run both in the same shell. - -```bash -CH_POD="$(kubectl -n cpaas-system get pod \ - -l clickhouse.altinity.com/chi=cpaas-clickhouse \ - -o jsonpath='{.items[0].metadata.name}')" -LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ - -o jsonpath='{.data.password}' | base64 -d)" -echo "platform-logging password: $LOG_PASSWORD" + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # From 4.1 ``` -### 3.6 Verify - -Run these in the same shell, so that `$CH_POD` and `$LOG_PASSWORD` from 3.5 are still set: +Save the YAML as `cpaas-clickhouse.yaml` and apply it: ```bash -kubectl -n cpaas-system exec "$CH_POD" -- \ - clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ - --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" - -kubectl -n cpaas-system exec "$CH_POD" -- \ - clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ - --query "DROP TABLE observability.__perm_check" +kubectl apply -f cpaas-clickhouse.yaml ``` -Both commands must succeed; a failed create means the account cannot manage the schema. - -Record these values: - -| Value | Where to read it | -| --- | --- | -| Endpoint | The cluster Service that exposes `8123`, listed by `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse` | -| Cluster name | `spec.configuration.clusters[].name` | -| Database | `spec.configuration.settings.default_database` | -| Shards / replicas | `shardsCount` / `replicasCount` | -| User / password | The account created above | - -## Step 4: Create the Kafka Service - -### 4.1 Create the SASL password Secret - -The password must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts. +`default_database: observability` makes ClickHouse create the `observability` database at startup, so there is no database to create here. -```bash -kubectl -n cpaas-system create secret generic platform-logging-password \ - --from-literal=password="$(openssl rand -hex 16)" -``` +#### 4.4.2 Single node (evaluation only) -### 4.2 Create the broker cluster +The Keeper runs inside the ClickHouse pod through the `keeper_server/*` settings. ```yaml -apiVersion: middleware.alauda.io/v1 -kind: RdsKafka +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation metadata: - name: cpaas-kafka + name: cpaas-clickhouse namespace: cpaas-system spec: - mode: KRaft - version: 4.2.0 # Minimum supported by the Alauda Kafka operator - replicas: 3 - resources: - limits: { cpu: "2", memory: 4Gi } # From the profile - requests: { cpu: 500m, memory: 2Gi } - storage: - size: 200Gi - class: - deleteClaim: false - controller: - replicas: 3 - roles: ["controller"] # Required: without it the node pool is rejected - template: - pod: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/infra # The label you set in Step 1 - operator: Exists - tolerations: - - key: node-role.kubernetes.io/infra # The taint you set in Step 1 - operator: Exists - effect: NoSchedule - resources: - limits: { cpu: "1", memory: 2Gi } - requests: { cpu: 100m, memory: 512Mi } - storage: - size: 20Gi - class: - deleteClaim: false - kafka: - template: - pod: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/infra # The label you set in Step 1 - operator: Exists - tolerations: - - key: node-role.kubernetes.io/infra # The taint you set in Step 1 - operator: Exists - effect: NoSchedule - listeners: - plain: - authentication: - type: scram-sha-512 - tls: - authentication: - type: scram-sha-512 - authorization: - type: simple - config: - auto.create.topics.enable: "false" - default.replication.factor: "3" - min.insync.replicas: "2" - offsets.topic.replication.factor: "3" - transaction.state.log.replication.factor: "3" - transaction.state.log.min.isr: "2" - log.retention.hours: "48" - unclean.leader.election.enable: "false" - message.max.bytes: "10485760" - replica.fetch.max.bytes: "10485760" - socket.request.max.bytes: "104857600" - entityOperator: - topicOperator: {} # Required: it creates the topics in Step 4.5 - userOperator: {} # Required: it creates the SASL user in Step 4.4 -``` - -Save the YAML as `cpaas-kafka.yaml` and apply it: - -```bash -kubectl apply -f cpaas-kafka.yaml -``` - -The instance creates its claims immediately. They stay `Pending` until you reserve volumes for them, because the broker claim names contain a hash that is generated per instance, which is why the volumes cannot be prepared up front: - -```bash -kubectl -n cpaas-system get pvc \ - -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' -``` + configuration: + users: + # The admin password comes from the Secret created above. + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # The account the logging components use. It is declared here, like the admin + # account, so it carries the privileges the components need without a separate + # set of GRANT statements. + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION -Copy the six names exactly — three `...-broker-...` and three `...-controller-...` — and create one pre-bound volume per claim. Set `capacity.storage` to the size the claim asks for, point `local.path` at that pod's directory, and pin the node: + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 -```yaml -apiVersion: v1 -kind: PersistentVolume -metadata: - name: cpaas-kafka-broker-0 -spec: - capacity: - storage: 200Gi - volumeMode: Filesystem - accessModes: - - ReadWriteOnce - persistentVolumeReclaimPolicy: Retain - storageClassName: cpaas-local-kafka - claimRef: # Reserve this volume for exactly this claim - apiVersion: v1 - kind: PersistentVolumeClaim - namespace: cpaas-system - name: data-cpaas-kafka-broker--0 - local: - path: /cpaas/data/kafka/broker-0 - nodeAffinity: - required: - nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/hostname - operator: In - values: [""] -``` + clusters: + - name: replicated # Reused in the connection Secret + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # From the profile: 1, 1, 2, or 3 + replicasCount: 1 # The example runs one pod; use 3 for three nodes and above -Save the YAML as `cpaas-kafka-volumes.yaml`, repeat the `PersistentVolume` part for all six claims, and apply it. The claims bind as soon as their volumes exist, and the brokers and controllers start: + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + # Self-observability system tables grow without bound and eventually fill the volume. + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + # Co-located Keeper for the single-node profile. For three nodes and above, + # use the manifest below instead, which runs Keeper in every ClickHouse pod. + keeper_server/tcp_port: "9181" + keeper_server/server_id: "1" + keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log + keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots + keeper_server/coordination_settings/operation_timeout_ms: "10000" + keeper_server/coordination_settings/session_timeout_ms: "30000" + keeper_server/raft_configuration/server/id: "1" + keeper_server/raft_configuration/server/hostname: localhost + keeper_server/raft_configuration/server/port: "9234" -```bash -kubectl apply -f cpaas-kafka-volumes.yaml -``` + zookeeper: + nodes: + - host: localhost + port: 9181 -Do not omit these settings: + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template -| Setting | Why it is required | -| --- | --- | -| `message.max.bytes: "10485760"` | Audit batches are about 1.1–1.5 MiB each. The Kafka default of 1 MiB rejects every audit batch. | -| `replica.fetch.max.bytes: "10485760"` | Must be at least `message.max.bytes`, otherwise replica synchronization stalls. | -| `auto.create.topics.enable: "false"` | A mistyped topic name must not be created automatically and silently collect data. | -| `entityOperator.topicOperator` / `userOperator` | Without them, the `RdsTopic` and `RdsKafkaUser` resources in the next steps are not applied to the brokers. | + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: keeper + containerPort: 9181 + - name: raft + containerPort: 9234 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse -### 4.3 Wait for the broker cluster + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP -```bash -kubectl -n cpaas-system get rdsKafka cpaas-kafka \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' -kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # From 4.1 ``` -Wait until the `Ready` condition is `True` and all broker pods are `Running`. The Kafka operator applies hard pod anti-affinity, so the three brokers must land on three different nodes: +Save the YAML as `cpaas-clickhouse.yaml` and apply it: ```bash -kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ - -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +kubectl apply -f cpaas-clickhouse.yaml ``` -Three distinct node names are required: hard anti-affinity needs at least three schedulable nodes. - -Confirm the required settings reached the brokers: +### 4.5 Wait for the cluster ```bash -KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ - -o jsonpath='{.items[0].metadata.name}')" +kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ + -o jsonpath='{.status.status}{"\n"}' # re-run until it reports: Completed -kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ - grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes" /tmp/strimzi.properties +kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse ``` -Both values must be `10485760`, otherwise stop. The broker applies these settings as static configuration, so `kafka-configs.sh --describe` does not show them. +All ClickHouse pods must be `Running` and ready, and every claim must be `Bound`. A StorageClass name that does not exist or cannot bind produces no pods and no error, so check the claims rather than the `ClickHouseInstallation` status alone. -### 4.4 Create the SASL user and its ACLs +#### 4.5.1 Three nodes and above (production) -The logging components use a single account with access to the three topics, the consumer groups, and the broker metadata. +Confirm the Keeper quorum before you continue. The Keeper runs inside the ClickHouse pods, so check one pod per replica: -```yaml -apiVersion: middleware.alauda.io/v1 -kind: RdsKafkaUser -metadata: - name: platform-logging - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - authentication: - type: scram-sha-512 - password: - valueFrom: - secretKeyRef: - name: platform-logging-password - key: password - authorization: - type: simple - acls: - # The three topics - - host: "*" - operation: All - resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } - - host: "*" - operation: All - resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } - - host: "*" - operation: All - resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } - # The consumer groups used by the log pipeline - - host: "*" - operation: All - resource: { type: group, name: alauda_log, patternType: literal } - - host: "*" - operation: All - resource: { type: group, name: alauda_event, patternType: literal } - - host: "*" - operation: All - resource: { type: group, name: alauda_audit, patternType: literal } - # The consumer group prefix used by LogForward - - host: "*" - operation: All - resource: { type: group, name: "logforward-", patternType: prefix } - # The consumer group prefix used by the log query service - - host: "*" - operation: All - resource: { type: group, name: "razor-", patternType: prefix } - # Broker metadata - - host: "*" - operation: All - resource: { type: cluster, name: kafka-cluster, patternType: literal } +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state ``` -Save the YAML as `platform-logging-user.yaml` and apply it: +The three lines must report one `leader` and two `follower`. Also confirm that ClickHouse reads the quorum through the Service from 4.3.1: ```bash -kubectl apply -f platform-logging-user.yaml +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# expect: cpaas-clickhouse-keeper 9181 ``` -All nine entries are required. `operation: All` covers the read and describe permissions these entries need. Verify: +#### 4.5.2 Single node (evaluation only) + +There is one Keeper member, so it reports `standalone` rather than a leader and followers. Confirm its state and the address it listens on: ```bash -kubectl -n cpaas-system get rdskafkauser platform-logging \ - -o jsonpath='{.status.phase}{"\n"}' # expect: Active -kubectl -n cpaas-system get secret platform-logging -``` +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +# expect: standalone -The group names use underscores (`alauda_log`), while the topic names use uppercase letters and underscores (`ALAUDA_LOG_TOPIC`). +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# expect: localhost 9181 +``` -### 4.5 Create the three topics +### 4.6 Read the logging account password -```yaml -apiVersion: middleware.alauda.io/v1 -kind: RdsTopic -metadata: - name: alauda-log-topic - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - topicName: ALAUDA_LOG_TOPIC # Broker-side name; must match the ACL and the connection Secret - partitions: 30 # Upper bound for consumer parallelism - replicas: 3 - config: - retention.ms: "172800000" # 48 hours - segment.bytes: "1073741824" - min.insync.replicas: "2" - compression.type: producer - max.message.bytes: "10485760" ---- -apiVersion: middleware.alauda.io/v1 -kind: RdsTopic -metadata: - name: alauda-event-topic - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - topicName: ALAUDA_EVENT_TOPIC - partitions: 30 - replicas: 3 - config: - retention.ms: "172800000" - segment.bytes: "1073741824" - min.insync.replicas: "2" - compression.type: producer - max.message.bytes: "10485760" ---- -apiVersion: middleware.alauda.io/v1 -kind: RdsTopic -metadata: - name: alauda-audit-topic - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - topicName: ALAUDA_AUDIT_TOPIC - partitions: 30 - replicas: 3 - config: - retention.ms: "172800000" - segment.bytes: "1073741824" - min.insync.replicas: "2" - compression.type: producer - max.message.bytes: "10485760" -``` +The account is declared in the `ClickHouseInstallation` you applied in 4.4, so there is nothing to create here and no `GRANT` statements to run: like the `admin` account, it is a server-configuration user and carries the privileges the logging components need. -Save the YAML as `alauda-topics.yaml` and apply it: +Read its password for the connection details in Step 6. 4.7 reuses the `CH_POD` and `LOG_PASSWORD` variables, so run both in the same shell. ```bash -kubectl apply -f alauda-topics.yaml +CH_POD="$(kubectl -n cpaas-system get pod \ + -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o jsonpath='{.items[0].metadata.name}')" +LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "platform-logging password: $LOG_PASSWORD" ``` -The resource name must be a valid DNS name. `spec.topicName` is the broker-side name and must match the ACL entries above. - -The example uses a 48-hour retention. Time-based retention alone does not cap disk usage: a burst of traffic can fill the broker volume before the window expires. Either size the broker volumes for the peak rate over the retention window, or add `retention.bytes` to each topic. `retention.bytes` applies per partition, so the broker volumes must hold `partitions × retention.bytes`. +### 4.7 Verify -### 4.6 Verify the Kafka service end to end - -Build the client properties inside the broker pod from the user Secret. The broker listeners require SASL, so every command below needs them. +Run these in the same shell, so that `$CH_POD` and `$LOG_PASSWORD` from 4.6 are still set: ```bash -KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ - -o jsonpath='{.items[0].metadata.name}')" +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" -kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ - sh -c 'cat > /tmp/logging-client.properties' <-kafka-bootstrap.cpaas-system.svc:9093` for SASL over TLS, or `:9092` for SASL without TLS | -| Cluster name | `metadata.name` of the `RdsKafka` resource | -| User / password | The `RdsKafkaUser` name and its password | -| Topics | `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, `ALAUDA_AUDIT_TOPIC` | -| CA certificate | Only for the TLS listener on `9093`. The Kafka operator publishes it in `cpaas-system` as a Secret whose name ends with `-cluster-ca-cert` | +1. Create the data directory on every node that will run an OpenSearch pod, and set its ownership (uid 1000). On Alauda OS nodes replace `/cpaas` with `/var/cpaas`: + + ```bash + sudo mkdir -p /cpaas/data/opensearch + sudo chown -R 1000:1000 /cpaas/data/opensearch + ``` -## Step 5: Create the OpenSearch Cluster +2. Make sure `vm.max_map_count` is at least `262144`. The OpenSearch operator sets it from an init container, so nothing is required unless your cluster enforces restricted Pod Security Admission, in which case that init container cannot run and you must set it on every node yourself: -Skip this step when the target is ClickHouse. + ```bash + sudo sysctl -w vm.max_map_count=262144 + echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf + ``` -### 5.1 Create the cluster +3. With static local volumes, pre-create one pre-bound PV per node using the shape from Step 1, with `claimRef` set to `data---`. For the example below that is `data-cpaas-opensearch-masters-0` to `data-cpaas-opensearch-masters-2` and `data-cpaas-opensearch-data-0` to `data-cpaas-opensearch-data-4`. With a dynamic provisioner, skip this. -Set the node pools from the Step 0 profile. The example is 3 + 5: three master nodes and five data nodes. For the small-scale profiles, use a single pool with `roles: [cluster_manager, data]` and `replicas: 3` or `5`. +Save the YAML as `cpaas-opensearch.yaml` and apply it: ```yaml apiVersion: opensearch.opster.io/v1 @@ -1173,7 +1400,7 @@ spec: replicas: 0 ``` -Save the YAML as `cpaas-opensearch.yaml` and apply it: +Apply it: ```bash kubectl apply -f cpaas-opensearch.yaml @@ -1305,44 +1532,74 @@ curl -sk -u "admin:" -X DELETE "$OS/_index_template/perm-check" curl -sk -u "admin:" -X DELETE "$OS/log-perm-check" ``` -Record these values: - -| Value | Where to read it | -| --- | --- | -| Endpoint | The service address or load balancer, for example `https://:9200` | -| User / password | The account created above | -| CA certificate | Only when the endpoint uses a private CA. The operator creates it in `cpaas-system` as Secret `cpaas-opensearch-ca` | +Record the connection values in Step 6. The platform applies index templates with one shard and one replica. If your HA policy needs different values, apply a composable template with a higher priority and confirm the result with `GET /_index_template`. Existing indices keep the settings they were created with. ## Step 6: Record the Connection Details -Record every value below. +Record the values below. The middle column is the key in the connection Secret that Installation reads the value from. + +### ClickHouse + +| Value | Connection Secret key | Where to read it | +| --- | --- | --- | +| Endpoint | `platform-default-ch-conn` → `endpoint` | The cluster Service that exposes 8123, listed by `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse`. Use `https://` and port 8443 when TLS is enabled | +| Cluster name | `cluster` | `spec.configuration.clusters[0].name` of the `ClickHouseInstallation`, which is also the name used by `ON CLUSTER`. Default: `replicated` | +| Database | `database` | `spec.configuration.settings.default_database`. Default: `observability` | +| Shards / replicas | Not in the Secret: `externalStorage.shards` / `replicas` in the `PlatformLogForward` | `spec.configuration.clusters[0].layout.shardsCount` / `replicasCount` | +| User | `username` | The logging account declared in the `ClickHouseInstallation`. Default: `platform-logging` | +| Password | `password` | `kubectl -n cpaas-system get secret clickhouse-platform-logging-password -o jsonpath='{.data.password}' \| base64 -d` | +| CA certificate | `tls.ca` | Only when the endpoint uses HTTPS with a private CA | + +### OpenSearch + +| Value | Connection Secret key | Where to read it | +| --- | --- | --- | +| Endpoint | `platform-default-os-conn` → `endpoints` | The service address or load balancer, for example `https://:9200` or `https://cpaas-opensearch.cpaas-system.svc:9200`. Put the highly available address first | +| User | `username` | The account created in 5.3. Default: `platform-logging` | +| Password | `password` | The password you set for that account | -| Value | ClickHouse | OpenSearch | Kafka | -| --- | --- | --- | --- | -| Endpoint | Required | Required | Required | -| Cluster name | Required | — | Required | -| Database | Required | — | — | -| Shards / replicas | Required | — | — | -| User | Required | Required | Required | -| Password | Required | Required | Required | -| Topics | — | — | Required | -| CA certificate | When the endpoint uses a private CA | When the endpoint uses a private CA (used for historical data migration) | When the endpoint uses a private CA | +### Kafka + +| Value | Connection Secret key | Where to read it | +| --- | --- | --- | +| Bootstrap address | `platform-default-mq-conn` → `bootstrap` | `-kafka-bootstrap.cpaas-system.svc:9093` for SASL over TLS, or `:9092` for SASL without TLS | +| Cluster name | `kafkaClusterName` | `metadata.name` of the `RdsKafka` resource. Default: `cpaas-kafka` | +| User | `username` | The `RdsKafkaUser` name. Default: `platform-logging` | +| Password | `password` | `kubectl -n cpaas-system get secret platform-logging-password -o jsonpath='{.data.password}' \| base64 -d`. At least 32 characters on Alauda OS nodes or other FIPS-enabled hosts | +| Topics | `topics.log` / `topics.event` / `topics.audit` | `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, `ALAUDA_AUDIT_TOPIC` | +| SASL mechanism | `sasl_mechanism` | `SCRAM-SHA-512` unless the broker uses another mechanism | +| CA certificate | `tls.ca` | Only for the TLS listener on 9093: `kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' \| base64 -d` | ## Environment Checklist +### Common + | Check | Expected | | --- | --- | -| Nodes and disks | Dedicated nodes labelled and tainted, SSD mounted, directories created with the right ownership, and the manifests tolerate that taint | -| StorageClass | Exists and binds the volumes the storage cluster uses | -| Operators | ClickHouse, Kafka, and (for OpenSearch) OpenSearch operators are ready, their CRDs exist, and they watch `cpaas-system` | -| ClickHouse | `status.status` is `Completed`, all pods ready, the logging account can create and drop a table. For three nodes and above, the Keeper quorum reports one leader and two followers | -| OpenSearch | Cluster health is `green`, the logging account can manage an index template, and `analysis-ik` is listed on every node and segments Chinese text when you enabled it | -| Kafka brokers | Cluster ready, `message.max.bytes` and `replica.fetch.max.bytes` are both `10485760` | +| Nodes and disks | Dedicated nodes labelled and tainted, SSD mounted for persistence, and the manifests tolerate that taint | +| StorageClass | Exists for every component you install, and binds the volumes it uses | +| Operators | The ClickHouse or OpenSearch operator and the Kafka operator are ready, their CRDs exist, and they watch `cpaas-system` | +| Kafka brokers | Cluster ready, `message.max.bytes` and `replica.fetch.max.bytes` are both `10485760`, and `log.retention.bytes` matches the topic cap | | Kafka user and ACLs | `RdsKafkaUser` is `Active`, all nine ACL entries are present | -| Kafka topics | The three topics exist on the brokers with the intended partitions and retention | +| Kafka topics | The three topics exist on the brokers with the intended partitions, `retention.bytes`, and `segment.bytes` | | Kafka connectivity | A record produced and consumed with the logging account | -| Values recorded | Endpoint, cluster, database, topology, credentials, topics, and CA certificates are all captured | + +### ClickHouse + +| Check | Expected | +| --- | --- | +| ClickHouse | `status.status` is `Completed`, all pods ready, and the logging account can create and drop a table. Three nodes and above: the Keeper quorum reports one leader and two followers. Single node: the single Keeper member reports `standalone` | +| Keeper Service | Three nodes and above: the headless Service from 4.3.1 selects the ready pods, and `system.zookeeper_connection` points at it. Single node: no Service is created, and `system.zookeeper_connection` points at `localhost` | +| Values recorded | Endpoint, cluster name, database, shards / replicas, user, password, and the CA certificate when the endpoint uses HTTPS | + +### OpenSearch + +| Check | Expected | +| --- | --- | +| OpenSearch | Cluster health is `green`, and the logging account can manage an index template | +| Chinese analyzer | `analysis-ik` is listed on every node and segments Chinese text when you enabled it | +| Values recorded | Endpoint, user, and password | Fix any failed check before the logging components connect to this storage. diff --git a/docs/zh/install/index.mdx b/docs/zh/install/index.mdx index 90208e3..2deacb7 100644 --- a/docs/zh/install/index.mdx +++ b/docs/zh/install/index.mdx @@ -1,165 +1,338 @@ --- weight: 14 -sourceSHA: f09d3facc79a57b4271628a04fea25c43cf9d09f43d5a42269d32c096c78b3d6 +sourceSHA: e33ff613ceb3b97d477085e52ccc4b46a9bcfa778b9d99c0e7b349057c32b24c --- # 安装 -本章把日志组件安装到由你自行运维的存储与消息队列上：一套 ClickHouse 或 OpenSearch 集群，以及一个 Kafka 服务。请先完成它们的准备，包括账号、ACL 与 topic： +本章把日志组件安装到[环境准备](../prepare/index.mdx)中创建的 ClickHouse 或 OpenSearch 3.7.0 集群和 Kafka 服务上。 -## 先准备存储 +无论使用 ClickHouse 还是 OpenSearch，步骤 1 都需要执行。然后执行步骤 2（ClickHouse）或步骤 3（OpenSearch）其中之一：两者互斥，各自创建连接 Secret 和 `PlatformLogForward`。步骤 4 对两种存储都需要执行，用于在 `PlatformLogForward` 就绪后部署两个集群插件。 -请先创建目标集群与 Kafka 服务，包括它们的账号、ACL 与 topic，并记录连接信息： +## 步骤 1：创建 Kafka 连接 Secret -- [环境准备](https://docs.alauda.cn/logging-service/4.3/prepare/index.html) - -在那里记录下来的连接信息，就是下面这些 Secret 的输入。 - -## 步骤 1：创建连接 Secret - -**在运行日志组件的集群上执行。** - -在 `cpaas-system` 中创建目标存储的 Secret 和 Kafka 服务的 Secret。请只创建与你的目标类型对应的那个存储 Secret。 - -### 目标 OpenSearch +**在工作负载集群执行。** ```yaml apiVersion: v1 kind: Secret metadata: - name: platform-default-os-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 + name: platform-default-mq-conn namespace: cpaas-system type: Opaque stringData: - endpoints: "https://:9200" # 必填；逗号分隔的 HTTP(S) 地址；请把高可用协调节点或负载均衡地址放在最前面 - username: "" # 目标端允许匿名访问时可省略 - password: "" # 目标端允许匿名访问时可省略 ---- -apiVersion: v1 -kind: Secret -metadata: - name: platform-default-mq-conn # Kafka 连接 Secret 名称，创建 PlatformLogForward 时引用 - namespace: cpaas-system -type: Opaque -stringData: - bootstrap: "" # 必填；Kafka 地址，host:port 形式，逗号分隔 - kafkaClusterName: "" # 必填；Kafka broker 资源名称，必须与实际名称一致 - username: "" # 必填；Kafka 用户名 - password: "" # 必填；在 Alauda OS 节点或其他启用 FIPS 的主机上至少 32 个字符 - sasl_mechanism: "SCRAM-SHA-512" # 可选，默认 SCRAM-SHA-512 - topics.log: "ALAUDA_LOG_TOPIC" # 可选，日志 topic 名，默认 ALAUDA_LOG_TOPIC - topics.event: "ALAUDA_EVENT_TOPIC" # 可选，事件 topic 名，默认 ALAUDA_EVENT_TOPIC - topics.audit: "ALAUDA_AUDIT_TOPIC" # 可选，审计 topic 名，默认 ALAUDA_AUDIT_TOPIC - tls.ca: |- # Kafka 使用 TLS 且证书不被系统信任时必填 + # 必填。Kafka 地址，格式为 host:port，多个 broker 用逗号分隔。 + # 从环境准备的步骤 3 中创建的服务读取。 + # 默认值：cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093（SASL over TLS） + # 或 cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9092（SASL 不加密） + bootstrap: "cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093" + # 必填。环境准备中 RdsKafka 资源的 metadata.name。 + kafkaClusterName: "cpaas-kafka" + # 必填。环境准备中 RdsKafkaUser 的名称。 + username: "platform-logging" + # 必填。你在 secret/platform-logging-password 的 password 字段中设置的密码。 + # 在 Alauda OS 节点或其他启用 FIPS 的主机上必须至少 32 个字符。 + password: "" + # 选填。默认 SCRAM-SHA-512。可选值：SCRAM-SHA-512、SCRAM-SHA-256、PLAIN。 + sasl_mechanism: "SCRAM-SHA-512" + # 选填。topic 名称，默认值如下，必须与环境准备的步骤 3 中创建的 topic 一致。 + topics.log: "ALAUDA_LOG_TOPIC" + topics.event: "ALAUDA_EVENT_TOPIC" + topics.audit: "ALAUDA_AUDIT_TOPIC" + # 选填。仅当监听器启用 TLS 且平台不信任其证书时需要。 + # 从 secret/-cluster-ca-cert 的 ca.crt 字段读取 PEM 内容。 + # 使用明文监听器（:9092）或证书由系统信任时，整段删除本字段。 + tls.ca: |- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ``` -### 目标 ClickHouse +需要读取的值： + +```bash +# Kafka 密码 +kubectl -n cpaas-system get secret platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d + +# Kafka 集群服务（bootstrap 地址） +kubectl -n cpaas-system get svc cpaas-kafka-kafka-bootstrap + +# Kafka CA，仅当监听器启用 TLS 且 CA 不被系统信任时需要 +kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert \ + -o jsonpath='{.data.ca\.crt}' | base64 -d +``` + +将 YAML 保存为 `platform-default-mq-conn.yaml` 并应用： + +```bash +kubectl apply -f platform-default-mq-conn.yaml +``` + +不要设置 `tls.insecure_skip_verify: "true"`；使用明文监听器时不填 `tls.ca` 即可。 + +## 步骤 2：ClickHouse：安装日志组件 + +本步骤只适用于 ClickHouse。使用 OpenSearch 时，请改用步骤 3。 + +### 2.1 创建 ClickHouse 连接 Secret + +**在工作负载集群执行。** ```yaml apiVersion: v1 kind: Secret metadata: - name: platform-default-ch-conn # 连接 Secret 名称，创建 PlatformLogForward 时引用 + name: platform-default-ch-conn namespace: cpaas-system type: Opaque stringData: - endpoint: "https://:8443" # 必填；ClickHouse 地址，包含协议和端口 - cluster: "replicated" # 必须与 ClickHouse 集群名一致；ACP 基线默认为 replicated - database: "observability" # 目标数据库名，默认 observability - username: "" # ClickHouse 用户名 - password: "" # ClickHouse 密码 - tls.ca: |- # 目标端使用 HTTPS 私有 CA 时必填 + # 必填。ClickHouse HTTP 地址，包含协议和端口：HTTP 为 8123，HTTPS 为 8443。 + # 从环境准备的步骤 4 中创建的集群 Service 读取： + # kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + endpoint: "http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123" + # 必须与 ClickHouseInstallation 的 spec.configuration.clusters[0].name + # 以及 ON CLUSTER 使用的集群名一致。ACP 基线默认值：replicated。 + cluster: "replicated" + # ClickHouse 数据库。默认值：observability（来自 ClickHouseInstallation）。 + database: "observability" + # ClickHouseInstallation 中声明的日志账号。默认值：platform-logging。 + username: "platform-logging" + # 该账号的密码，存放在 secret/clickhouse-platform-logging-password 中。 + password: "" + # 选填。仅当 endpoint 使用 HTTPS 且为自签 CA 时需要。 + # 使用明文 HTTP（:8123）时，整段删除本字段。 + tls.ca: |- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ``` -`endpoints` 支持以逗号分隔的多个 HTTP(S) 地址。部分数据链路只使用第一个地址，因此请把高可用的负载均衡器或 coordinator 地址放在最前面，不要使用单个数据节点，也不要在第一个 URL 前留空格。 +需要读取的值： + +```bash +# ClickHouse 服务与端口 +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + +# 集群名、数据库、shard / replica 数量 +kubectl -n cpaas-system get chi cpaas-clickhouse \ + -o jsonpath='{.spec.configuration.clusters[0].name}{"\t"}{.spec.configuration.settings.default_database}{"\t"}{.spec.configuration.clusters[0].layout.shardsCount}{"\t"}{.spec.configuration.clusters[0].layout.replicasCount}{"\n"}' + +# 日志账号密码 +kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d +``` -OpenSearch 的 Secret 不包含 TLS 相关字段。平台连接 OpenSearch 时关闭了证书校验，因此使用私有 CA 的地址不需要在这里填写任何证书;地址使用 `https://` 即可建立连接，与签发方无关。 +将 YAML 保存为 `platform-default-ch-conn.yaml` 并应用： -ClickHouse 请勿设置 `tls.insecure_skip_verify: "true"`，而应提供 `tls.ca`，以便平台校验目标端。 +```bash +kubectl apply -f platform-default-ch-conn.yaml +``` -## 步骤 2：创建 PlatformLogForward +不要设置 `tls.insecure_skip_verify: "true"`。endpoint 使用 HTTPS 时必须提供 `tls.ca` 供平台校验服务端证书；使用明文 HTTP 时不填 `tls.ca`。 -**在运行日志组件的集群上执行。** +### 2.2 创建 PlatformLogForward -为你的目标类型创建一个 `PlatformLogForward`。 +**在工作负载集群执行。** -### 目标 OpenSearch +使用 ClickHouse 时必须填写 `output.type`。 ```yaml apiVersion: log.alauda.io/v1alpha1 kind: PlatformLogForward metadata: - name: platform-default # 固定的集群单例名称，请勿修改 + name: platform-default # 集群级单例名称，固定不变 spec: - installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + installMode: Fresh # 固定为 Fresh，不要改成 Adopt + output: + type: clickhouse # 使用 ClickHouse 时必填 externalStorage: - type: opensearch # 目标存储类型 + type: clickhouse # 存储类型 + shards: 1 # 按下面的规格表填写：单节点 1、六节点 2、九节点 3 + replicas: 1 # 按下面的规格表填写：单节点 1、三节点及以上 3 secretRef: - name: platform-default-os-conn # 步骤 1 创建的目标存储连接 Secret + name: platform-default-ch-conn # 步骤 2.1 创建的连接 Secret namespace: cpaas-system externalMessageQueue: type: kafka # 消息队列类型，目前仅支持 kafka secretRef: - name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret namespace: cpaas-system ``` -### 目标 ClickHouse +`externalStorage.shards` 和 `externalStorage.replicas` 必须与环境准备中实际部署的 ClickHouse 规格一致： + +| 规格 | `shards` | `replicas` | +| --- | --- | --- | +| 单节点（仅用于验证环境） | 1 | 1 | +| 三节点 | 1 | 3 | +| 六节点 | 2 | 3 | +| 九节点 | 3 | 3 | + +多 shard 或多副本场景填错会导致 ClickHouse 的部分拓扑用不上。 + +`PlatformLogForward` 是集群级资源，不要添加 `metadata.namespace`；`secretRef` 内的 `namespace` 仍用于定位 `cpaas-system` 中的连接 Secret。CRD 默认 `aggregateVector.replicas: 3`、`razor.replicas: 2`，如果容量或部署规划要求不同副本数，请显式设置。这两个是日志组件自身的副本数，与 ClickHouse 规格无关：单节点 ClickHouse 也照常使用它们。 + +将 YAML 保存为 `platform-log-forward.yaml` 并应用： + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +### 2.3 校验 + +观察状态直到完成，按 `Ctrl+C` 退出： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 变为 `Ready`、`Ready` 列为 `True` 即为完成。查看状态条件可用于跟踪进度或排查： -目标为 ClickHouse 时必须填写 `output.type` 字段。 +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +`PlatformLogForward` 已就绪。请继续执行步骤 4 部署集群插件，然后在步骤 4.3 做端到端数据校验。 + +## 步骤 3：OpenSearch：安装日志组件 + +本步骤只适用于 OpenSearch。使用 ClickHouse 时，请改用步骤 2。 + +### 3.1 创建 OpenSearch 连接 Secret + +**在工作负载集群执行。** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn + namespace: cpaas-system +type: Opaque +stringData: + # 必填。逗号分隔的 HTTP(S) URL，请把高可用 coordinator 或负载均衡地址放在第一个， + # 部分数据路径只使用第一个地址。从环境准备的步骤 5 中创建的 OpenSearch 服务读取， + # 例如 https://cpaas-opensearch.cpaas-system.svc:9200 或其前置负载均衡地址。 + # 第一个地址前不要有空格。 + endpoints: "https://:9200" + # 环境准备的步骤 5 中创建的账号，默认值：platform-logging。 + # 集群允许匿名访问时这两个字段都可以省略。 + username: "" + password: "" +``` + +需要读取的值： + +```bash +# OpenSearch 服务与端口 +kubectl -n cpaas-system get svc cpaas-opensearch + +# 集群健康状态 +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' +``` + +将 YAML 保存为 `platform-default-os-conn.yaml` 并应用： + +```bash +kubectl apply -f platform-default-os-conn.yaml +``` + +本 Secret 没有 `tls.ca` 字段：平台连接 OpenSearch 时关闭证书校验，因此使用自签 CA 的地址也无需填写。`endpoints` 使用 `https://` 即可，无论签发者是谁都能建立连接。 + +### 3.2 创建 PlatformLogForward + +**在工作负载集群执行。** ```yaml apiVersion: log.alauda.io/v1alpha1 kind: PlatformLogForward metadata: - name: platform-default # 固定的集群单例名称，请勿修改 + name: platform-default # 集群级单例名称，固定不变 spec: - installMode: Fresh # 始终为 Fresh，请勿改为 Adopt - output: - type: clickhouse # 目标为 ClickHouse 时必填 + installMode: Fresh # 固定为 Fresh，不要改成 Adopt externalStorage: - type: clickhouse # 目标存储类型 - shards: 1 # 目标 ClickHouse 的实际分片数 - replicas: 1 # 目标 ClickHouse 的实际副本数 + type: opensearch # 存储类型 secretRef: - name: platform-default-ch-conn # 步骤 1 创建的目标存储连接 Secret + name: platform-default-os-conn # 步骤 3.1 创建的连接 Secret namespace: cpaas-system externalMessageQueue: type: kafka # 消息队列类型，目前仅支持 kafka secretRef: - name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret，topic 名来自该 Secret + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret namespace: cpaas-system ``` -`externalStorage.shards` 与 `externalStorage.replicas` 必须与 ClickHouse 实际拓扑一致。两者默认都是 `1`;多分片或多副本部署中填错会导致部分拓扑不被使用。 +`PlatformLogForward` 是集群级资源，不要添加 `metadata.namespace`；`secretRef` 内的 `namespace` 仍用于定位 `cpaas-system` 中的连接 Secret。CRD 默认 `aggregateVector.replicas: 3`、`razor.replicas: 2`，如果容量或部署规划要求不同副本数，请显式设置。 -`PlatformLogForward` 是集群级资源，请不要为它添加 `metadata.namespace`;`secretRef` 里的 `namespace` 字段仍用于指向 `cpaas-system` 中的连接 Secret。CRD 默认值为 `aggregateVector.replicas: 3` 和 `razor.replicas: 2`;如果容量或调度规划需要不同的副本数，请显式设置。 - -请把 YAML 保存为 `platform-log-forward.yaml` 并应用： +将 YAML 保存为 `platform-log-forward.yaml` 并应用： ```bash kubectl apply -f platform-log-forward.yaml ``` -## 步骤 3：校验 +### 3.3 校验 -观察状态直到完成，按 `Ctrl+C` 停止： +观察状态直到完成，按 `Ctrl+C` 退出： ```bash kubectl get platformlogforward platform-default -w ``` -`Phase` 列会变为 `Ready`，`Ready` 列会变为 `True`。需要跟踪进度或排查问题时，读取 status conditions： +`Phase` 变为 `Ready`、`Ready` 列为 `True` 即为完成。查看状态条件可用于跟踪进度或排查： ```bash kubectl get platformlogforward platform-default \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` -随后产生或找到新的日志、事件与审计记录，确认可以从目标存储中查询到它们。 +`PlatformLogForward` 已就绪。请继续执行步骤 4 部署集群插件，然后在步骤 4.3 做端到端数据校验。 + +## 步骤 4：部署集群插件 + +**请在 `PlatformLogForward/platform-default` 为 `Ready=True` 后执行。** + +无论使用 ClickHouse 还是 OpenSearch，本步骤都需要执行。`PlatformLogForward` 只安装存储与转发数据链路，不会安装下面两个集群插件。 + +### 4.1 部署 Log Essentials + +**在 global 集群执行。** + +1. 打开 **Marketplace** > **Cluster Plugins**，选择 `global`。 +2. 使用默认配置安装 **Alauda Container Platform Log Essentials**。 +3. 校验插件运行状态： + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=log-api + ``` + +`STATUS` 列必须为 `Running`。 + +### 4.2 部署 Log Collector + +请在每个需要采集日志、事件和审计数据的集群上执行本步骤。如果也需要采集 global 集群自身的数据，请在 `global` 上安装。 + +1. 打开 **Marketplace** > **Cluster Plugins**，选择要安装的集群。 +2. 安装 **Alauda Container Platform Log Collector**，并填写： + +| 字段 | 取值 | +| --- | --- | +| **日志存储插件** | **Standard**。这是由 `log-storage-operator` 管理的数据链路，ClickHouse 和 OpenSearch 都选这一项。不要选择旧的 `ClickHouse` 或 `ElasticSearch` 选项。 | +| **存储集群名称** | 填写 `PlatformLogForward/platform-default` 为 `Ready` 的集群名称。采集器和存储在同一个集群时，填写该集群名称。该字段是文本输入框，`log-storage-operator` 场景不会自动发现存储集群。 | +| **日志采集器存储路径** | 填写采集器本地工作数据的绝对路径。传统操作系统使用 `/cpaas`；Alauda OS 使用 `/var/cpaas` 下可写的路径，例如 `/var/cpaas`。 | +| **挂载路径** | 选填。添加采集器需要读取日志文件的节点绝对路径。 | +| **Audit**、**Event**、**Kubernetes**、**Platform**、**System**、**Workload** | 选择需要采集的日志类型。默认启用 Audit、Event、System 和 Workload，Kubernetes 和 Platform 默认关闭。 | + +3. 安装插件并校验运行状态： + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=logagent + ``` + +请在 global 集群执行校验命令。每个采集集群对应的行都必须显示 `STATUS=Running`。 + +### 4.3 校验数据链路 + +两个插件的状态都通过后，产生或查找新的日志、事件和审计数据，确认可以从 ClickHouse 或 OpenSearch 查询到。如果没有数据到达，请先检查 `Log Collector` 的 `ModuleInfo` 状态和 `PlatformLogForward` conditions，再修改资源。 diff --git a/docs/zh/prepare/index.mdx b/docs/zh/prepare/index.mdx index f089c03..7de5596 100644 --- a/docs/zh/prepare/index.mdx +++ b/docs/zh/prepare/index.mdx @@ -1,11 +1,11 @@ --- weight: 13 -sourceSHA: 78020a1b837c09abfd42a624d3d6a1fc12b64ac26c3152c44d72f724197548e8 +sourceSHA: 2beb29dea60f229162c315ea1be37441ca745c03c69edad76669447818d40776 --- # 环境准备 -本文说明如何搭建日志组件使用的 ClickHouse 或 OpenSearch 3.7.0 集群和 Kafka 服务，包括节点与磁盘、Operator、集群及其账号，以及 Kafka 的密码、用户、ACL 和 topic。 +本文介绍如何准备日志组件使用的 ClickHouse 或 OpenSearch 3.7.0 集群和 Kafka 服务。步骤 1 至 3 对 ClickHouse 和 OpenSearch 都适用；步骤 4 仅用于 ClickHouse，步骤 5 仅用于 OpenSearch，请只执行你使用的那一步。 请按顺序执行并完成校验。 @@ -15,15 +15,15 @@ sourceSHA: 78020a1b837c09abfd42a624d3d6a1fc12b64ac26c3152c44d72f724197548e8 1. 你拥有运行日志组件的集群的管理员权限。 2. 已按步骤 1 规划好节点和磁盘。 -3. 平台市场中已上架以下 Operator 包：`clickhouse-operator`、Alauda Kafka Operator、`opensearch-operator`。 +3. 平台市场中已上架以下 Operator 包：`log-storage-operator`、`clickhouse-operator`、Alauda Kafka Operator、`opensearch-operator`。 请结合[日志组件容量规划](https://docs.alauda.cn/logging-service/4.3/architecture/capacity_planning.html)和下面的表格确定规格，并按[为日志存储规划基础设施节点](https://docs.alauda.cn/logging-service/4.3/how_to/infra_nodes.html)把工作负载放到独占节点上。 -本章所有命令都请在能访问该集群的 `kubectl` 主机上执行。每段 YAML 都需要先保存成文件再用 `kubectl apply -f` 应用；每段代码块下方都给出了文件名和对应的 apply 命令。 +本章所有命令都请在能访问该集群的 `kubectl` 主机上执行。每段 YAML 都需要先保存成文件再用 `kubectl apply -f` 应用；每一步都会给出文件名和对应的 apply 命令。 -## 步骤 0：选择目标类型和规格 +## 步骤 0：选择 ClickHouse 或 OpenSearch 及规格 -先确定目标类型（ClickHouse 或 OpenSearch）和规格。 +请先确定使用 ClickHouse 还是 OpenSearch，以及规格。两者都需要完成步骤 3 的 Kafka 服务；步骤 4 与步骤 5 互斥，只执行其中之一。 ### ClickHouse 规格 @@ -36,11 +36,11 @@ CPU 和内存是每个 ClickHouse Pod 的容器 limit。 | 六节点 | 6 | 2 分片 × 3 副本 | 4C | 8G | 40,000 logs/s | | 九节点 | 9 | 3 分片 × 3 副本 | 4C | 8G | 69,000 logs/s | -单节点规格仅用于验证环境。生产环境请从三节点规格起步；单个分片无法容纳数据时，再扩展到六节点或九节点。 +单节点规格仅用于验证环境，配合步骤 3 的单节点 Kafka 清单使用。生产环境请从三节点规格起步；单个分片无法容纳数据时，再扩展到六节点或九节点。 ### Kafka -请准备三个 broker，每个 limit 为 2C/4G，另外还有步骤 4 清单中三个 limit 为 1C/2G 的 controller。broker 存储量按保留时长和吞吐规划。 +请准备三个 broker，每个 limit 为 2C/4G，另外还有步骤 3 清单中三个 limit 为 1C/2G 的 controller。broker 存储量按保留时长和吞吐规划。单节点验证规格只运行一个 broker 和一个 controller，limit 与上面相同。 ### OpenSearch 规格 @@ -53,7 +53,7 @@ CPU 和内存是每个节点的 limit。 | 大规格 | 3 + 5 | 3 个 master，5 个 data | master 2C / data 8C | master 4G / data 16G | 25,000 logs/s | | 大规格 | 3 + 7 | 3 个 master，7 个 data | master 2C / data 8C | master 4G / data 16G | 30,000 logs/s | -不要低于最小规格，当单个节点池无法承载数据量时，请改用大规格。如果实际存储低于 6,000 IOPS 和 250 MB/s 读写，请上调规格。 +OpenSearch 最小规格为三节点，不支持单节点部署。不要低于最小规格，当单个节点池无法承载数据量时，请改用大规格。如果实际存储低于 6,000 IOPS 和 250 MB/s 读写，请上调规格。 ### 磁盘 @@ -67,52 +67,22 @@ CPU 和内存是每个节点的 limit。 - 使用传统操作系统布局时，使用 `/cpaas/data/...`。 - 在 Alauda OS 节点上只有 `/var/cpaas` 可写，因此使用 `/var/cpaas/data/...`。 4. 确保该路径在节点重新纳管后仍然保留。 -5. OpenSearch 需要把 `vm.max_map_count` 设置为不小于 `262144`。OpenSearch Operator 会通过 init 容器设置它，因此这里通常不用处理；只有集群启用了受限的 Pod Security Admission 时，该 init 容器无法设置，才需要在每个节点上手工设置： - - ```bash - sudo sysctl -w vm.max_map_count=262144 - echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf - ``` -6. 创建存储 Pod 使用的目录并设置属主。下面的示例使用传统布局；在 Alauda OS 节点上请把 `/cpaas` 换成 `/var/cpaas`。 - - ```bash - # ClickHouse 以 uid 101 运行 - sudo mkdir -p /cpaas/data/clickhouse - sudo chown -R 101:101 /cpaas/data/clickhouse - - # Kafka 以 uid 1001 运行；每个 Pod 一个目录，见下文 - sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-3 - sudo chown -R 1001:1001 /cpaas/data/kafka - - # OpenSearch 以 uid 1000 运行 - sudo mkdir -p /cpaas/data/opensearch - sudo chown -R 1000:1000 /cpaas/data/opensearch - ``` - -7. 确定存储卷的供给方式： +5. 确定存储卷的供给方式： | 方式 | 适用场景 | 需要做什么 | | --- | --- | --- | -| 静态本地卷 | 需要把每个 Pod 固定到指定节点，这也是基础设施节点方案通常采用的方式 | 创建一个不带 provisioner 的 StorageClass，并按预期的 Pod 数量预先创建 PV，每个 PV 通过 `nodeAffinity` 和 `local.path` 指向上面的目录 | +| 静态本地卷 | 需要把每个 Pod 固定到指定节点，这也是基础设施节点方案通常采用的方式 | 创建一个不带 provisioner 的 StorageClass，并按预期的 Pod 数量预先创建 PV，每个 PV 通过 `nodeAffinity` 和 `local.path` 指向该 Pod 的目录 | | 动态供给 | 平台已提供块存储 provisioner | 创建 StorageClass，由 PVC 动态绑定；确认 provisioner 支持 `ReadWriteOnce` 块卷并满足上述吞吐要求 | 使用静态本地卷时，请创建一个 StorageClass 并按每个 Pod 一个 PV 预先创建，同时**把每个 PV 预留给它自己的 PVC**。`local` 卷不能跟随 Pod 迁移：如果不做预留，某个 PVC 可能绑走为其他 Pod 或其他组件准备的卷；实例删除重建后，各 Pod 之间也可能互相绑错盘。用 `spec.claimRef` 预留之后，这个卷只会匹配它上面写明的那个 PVC。 -请为每个组件使用独立的 StorageClass。下面的示例创建 ClickHouse 那个，并为第一个 ClickHouse 副本预留一个卷。PV 数量等于你规划的 Pod 总数：ClickHouse 为 `shardsCount × replicasCount`，Kafka 为 `replicas + controller.replicas`，OpenSearch 为各节点池 `replicas` 之和。 - -ClickHouse 和 OpenSearch 的 PVC 名称是确定的，因此可以在实例创建之前就把卷预留好： - -| 组件 | 副本 `` 的 PVC 名称 | -| --- | --- | -| ClickHouse | `data-volumeclaim-template-chi-----0` | -| OpenSearch | `data---` | -| Kafka | `data--broker--`，需要先创建实例再读取名称，见步骤 4 | +请为每个组件使用独立的 StorageClass，避免某个组件的 PVC 绑走为其他组件准备的卷。PV 数量等于你规划的 Pod 总数：ClickHouse 为 `shardsCount × replicasCount`，Kafka 为 `replicas + controller.replicas`，OpenSearch 为各节点池 `replicas` 之和。各组件的清单写法相同，具体的 PVC 名称、目录和容量在对应步骤中给出（Kafka 见步骤 3，ClickHouse 见步骤 4，OpenSearch 见步骤 5），把它们填进下面的模板： ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: - name: cpaas-local-clickhouse + name: cpaas-local- labels: # 在项目命名空间中是必需的：没有这个授权， # pvc-validator 准入 webhook 会拒绝所有使用该 StorageClass 的 PVC。 @@ -125,22 +95,22 @@ allowVolumeExpansion: false apiVersion: v1 kind: PersistentVolume metadata: - name: cpaas-clickhouse-0 + name: spec: capacity: - storage: 200Gi + storage: volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain - storageClassName: cpaas-local-clickhouse + storageClassName: cpaas-local- claimRef: # 该卷只留给下面这个 PVC apiVersion: v1 kind: PersistentVolumeClaim namespace: cpaas-system - name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + name: local: - path: /cpaas/data/clickhouse + path: nodeAffinity: required: nodeSelectorTerms: @@ -150,313 +120,770 @@ spec: values: [""] ``` -请把 YAML 保存为 `local-storage.yaml` 并应用： - -```bash -kubectl apply -f local-storage.yaml -``` - -请为每个 Pod 重复 `PersistentVolume` 部分，`metadata.name` 和 `local.path` 各不相同，`values` 填该 Pod 所在节点的 IP。同一节点上跑多个同类 Pod 时，每个 Pod 用独立目录，例如 `/cpaas/data/clickhouse-0` 和 `/cpaas/data/clickhouse-1`，属主按步骤 6 设置。 +请为每个 Pod 重复 `PersistentVolume` 部分，`metadata.name` 和 `local.path` 各不相同，`values` 填该 Pod 所在节点的 IP。同一节点上跑多个同类 Pod 时，每个 Pod 用独立目录。 `local` 卷上的 `capacity.storage` 只是匹配用的元数据，不是配额：Pod 完全可以写满底层磁盘。请把它设成真实可用容量，并在存储侧同时配置保留策略。 ## 步骤 2：安装 Operator -请从平台市场安装这三个 Operator。下面创建的存储和消息资源都位于 `cpaas-system`，因此每个 Operator 都必须能协调该命名空间中的资源。 +请从平台市场安装所需 Operator。下面创建的 `ClickHouseInstallation`、`RdsKafka` 和 OpenSearch 资源，以及安装阶段创建的 `PlatformLogForward`，都位于 `cpaas-system`，因此每个 Operator 都必须能协调该命名空间中的资源。仅 ClickHouse 需要安装 ClickHouse Operator，仅 OpenSearch 需要安装 OpenSearch Operator；两种场景都需要 Kafka Operator 和 `log-storage-operator`。 | Operator | Subscription 所在命名空间 | Operator 必须 watch 的命名空间 | | --- | --- | --- | -| `clickhouse-operator` | `cpaas-system` | `cpaas-system` | +| `log-storage-operator` | `cpaas-system` | `cpaas-system` | +| `clickhouse-operator`（仅 ClickHouse） | `cpaas-system` | `cpaas-system` | | Alauda Kafka Operator（`strimzi-kafka-operator`） | `kafka-system` | 所有命名空间 | -| `opensearch-operator`（仅目标存储为 OpenSearch 时） | `opensearch-operator` | 所有命名空间 | +| `opensearch-operator`（仅 OpenSearch） | `opensearch-operator` | 所有命名空间 | - 只安装缺失的 Operator。如果集群上已经装了某一个（例如由更早的版本装入），请沿用已有的，不要再装第二份：同一个 Operator 的两份实例会同时写 `cpaas-system` 下的同一批资源。此时改为检查它的 watch 范围，必要时放开。 - 不要在 `cpaas-system` 中创建 OperatorGroup。平台已在其中创建了一个，再创建一个会导致平台拒绝该命名空间下的所有 Subscription，包括平台自己的。 - `kafka-system` 和 `opensearch-operator` 的 OperatorGroup 必须不包含 `spec.targetNamespaces`。如果已经存在一个只作用于自身命名空间的 OperatorGroup，请删除该字段，并等待 Operator Pod 重启。 - ```bash - kubectl -n kafka-system patch operatorgroup kafka-system \ - --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' - kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ - --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' - ``` +:::warning +Operator 如果未 watch `cpaas-system`，会静默忽略下面的资源：没有 status、没有事件、也没有 Pod。继续之前请确认 Deployment 已就绪，且 Kafka 和 OpenSearch 的 OperatorGroup 覆盖所有命名空间。 +::: + +### log-storage-operator + +两种场景都需要。安装阶段创建的 `PlatformLogForward` 由它提供。 + +```bash +kubectl get crd platformlogforwards.log.alauda.io logforwards.log.alauda.io +kubectl -n cpaas-system get deploy log-storage-operator-controller-manager +kubectl -n cpaas-system get sub log-storage-operator +``` -继续之前请校验每个 Operator： +### ClickHouse Operator + +仅在 ClickHouse 时需要。`cpaas-system` 中已有 OperatorGroup，无需额外操作。 ```bash -# ClickHouse kubectl get crd clickhouseinstallations.clickhouse.altinity.com kubectl -n cpaas-system get deploy clickhouse-operator +``` + +### Kafka Operator + +ClickHouse 和 OpenSearch 都需要。 + +```bash +kubectl -n kafka-system patch operatorgroup kafka-system \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' -# Kafka kubectl get crd rdskafkas.middleware.alauda.io kubectl -n kafka-system get deploy strimzi-cluster-operator +``` + +### OpenSearch Operator + +仅在 OpenSearch 时需要。 + +```bash +kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' -# OpenSearch（仅当目标存储为 OpenSearch 时） kubectl get crd opensearchclusters.opensearch.opster.io kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager ``` -:::warning -Operator 如果未 watch `cpaas-system`，会静默忽略下面的资源：没有 status、没有事件、也没有 Pod。继续之前请确认 Deployment 已就绪，且 Kafka 和 OpenSearch 的 OperatorGroup 覆盖所有命名空间。 -::: - -## 步骤 3：创建 ClickHouse 集群 - -目标存储为 OpenSearch 时请跳过本步骤。 +## 步骤 3：创建 Kafka 服务 -### 3.1 创建密码 Secret +### 3.1 创建 SASL 密码 Secret -实例定义了两个账号：用于管理的 `admin`，以及供日志组件使用的 `platform-logging`。两者的密码都来自 Secret，因此请在创建实例之前先把两个 Secret 建好。 +在 Alauda OS 节点或其他启用了 FIPS 的主机上，密码长度必须不少于 32 个字符。 ```bash -kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ - --from-literal=password="$(openssl rand -hex 16)" -kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ +kubectl -n cpaas-system create secret generic platform-logging-password \ --from-literal=password="$(openssl rand -hex 16)" ``` -### 3.2 创建 Keeper 客户端 Service（三节点及以上） +### 3.2 创建 broker 集群 -所有规格都需要 Keeper，单节点也一样，因为日志组件创建的是 `ReplicatedMergeTree` 表。单节点规格把 Keeper 运行在自己的 ClickHouse Pod 内，靠 3.3 中的 `keeper_server/*` 配置实现，因此单节点请跳过本步骤。 +请选择与步骤 0 中选定的规格对应的一份清单：三节点及以上是生产规格，单节点仅用于验证环境。两者的区别只有副本数和副本因子，不要混用。 -三节点及以上时，由 ClickHouse Pod 自身组成 Keeper 仲裁集群：每个 ClickHouse Pod 同时也是一个 Keeper 成员。ClickHouse 通过一个 headless Service 访问该仲裁集群，该 Service 会选中本次安装中所有就绪的 Pod。 +以下配置不能省略： -请把 YAML 保存为 `cpaas-clickhouse-keeper-service.yaml` 并应用： +| 配置项 | 为什么必须设置 | +| --- | --- | +| `message.max.bytes: "10485760"` | 审计数据的批大小约为 1.1–1.5 MiB，Kafka 默认的 1 MiB 会拒绝所有审计批次。 | +| `replica.fetch.max.bytes: "10485760"` | 必须不小于 `message.max.bytes`，否则副本同步会停滞。 | +| `auto.create.topics.enable: "false"` | 避免拼错的 topic 名称被自动创建并静默接收数据。 | +| `entityOperator.topicOperator` / `userOperator` | 缺少它们时，下一步中的 `RdsTopic` 和 `RdsKafkaUser` 不会生效到 broker。 | + +#### 3.2.1 三节点及以上（生产） + +在每台将运行 broker 或 controller 的节点上，为每个 Pod 创建一个目录并设置属主（uid 1001）。在 Alauda OS 节点上请把 `/cpaas` 换成 `/var/cpaas`： + +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/broker-1 /cpaas/data/kafka/broker-2 +sudo mkdir -p /cpaas/data/kafka/controller-0 /cpaas/data/kafka/controller-1 /cpaas/data/kafka/controller-2 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` ```yaml -apiVersion: v1 -kind: Service +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka metadata: - name: cpaas-clickhouse-keeper + name: cpaas-kafka namespace: cpaas-system spec: - clusterIP: None - type: ClusterIP - ports: - - name: keeper - port: 9181 - protocol: TCP - targetPort: 9181 - selector: - clickhouse.altinity.com/chi: cpaas-clickhouse - clickhouse.altinity.com/namespace: cpaas-system - clickhouse.altinity.com/ready: "yes" - clickhouse.altinity.com/role: keeper + mode: KRaft + version: 4.2.0 # Alauda Kafka Operator 支持的最低版本 + replicas: 3 + resources: + limits: { cpu: "2", memory: 4Gi } # 取自规格 + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + controller: + replicas: 3 + roles: ["controller"] # 必需：缺少时 node pool 会被拒绝 + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # 必需：由它创建 3.5 中的 topic + userOperator: {} # 必需：由它创建 3.4 中的 SASL 用户 ``` +请把 YAML 保存为 `cpaas-kafka.yaml` 并应用： + ```bash -kubectl apply -f cpaas-clickhouse-keeper-service.yaml +kubectl apply -f cpaas-kafka.yaml ``` -其中 `chi`、`namespace`、`ready` 三个标签由 Operator 打上，`role: keeper` 标签来自 3.3 的 pod template。 - -### 3.3 创建 ClickHouseInstallation +#### 3.2.2 单节点（仅用于验证环境） -cluster 名称固定为 `replicated`，需与日志组件保持一致；`shardsCount` 和 `replicasCount` 按步骤 0 的规格设置。下面两份清单请按你选的规格二选一执行。 - -请把 `` 替换为平台 middleware 包发布的 ClickHouse server 镜像，例如 `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`。 +只需要为唯一的 broker 和唯一的 controller 各创建一个目录： -**单节点。** Keeper 通过 `keeper_server/*` 配置运行在 ClickHouse Pod 内。 +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-0 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` ```yaml -apiVersion: clickhouse.altinity.com/v1 -kind: ClickHouseInstallation +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka metadata: - name: cpaas-clickhouse + name: cpaas-kafka namespace: cpaas-system spec: - configuration: - users: - # 管理员密码来自上面创建的 Secret - admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password - admin/networks/ip: - - "0.0.0.0/0" - - "::/0" - admin/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION - # 日志组件使用的账号。与 admin 一样在这里声明， - # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 - platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password - platform-logging/networks/ip: - - "0.0.0.0/0" - - "::/0" - platform-logging/profile: default - platform-logging/quota: default - platform-logging/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION + mode: KRaft + version: 4.2.0 # Alauda Kafka Operator 支持的最低版本 + replicas: 1 + resources: + limits: { cpu: "2", memory: 4Gi } # 取自规格 + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + controller: + replicas: 1 + roles: ["controller"] # 必需：缺少时 node pool 会被拒绝 + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "1" + min.insync.replicas: "1" + offsets.topic.replication.factor: "1" + transaction.state.log.replication.factor: "1" + transaction.state.log.min.isr: "1" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # 必需：由它创建 3.5 中的 topic + userOperator: {} # 必需：由它创建 3.4 中的 SASL 用户 +``` - profiles: - default/allow_nondeterministic_mutations: "1" - default/allow_unrestricted_reads_from_keeper: "1" - default/max_execution_time: 120 - default/max_estimated_execution_time: 120 +请把 YAML 保存为 `cpaas-kafka.yaml` 并应用： - clusters: - - name: replicated # 连接 Secret 中需要复用该名称 - templates: - podTemplate: pod-template - dataVolumeClaimTemplate: data-volumeclaim-template - layout: - shardsCount: 1 # 取自规格：1、1、2 或 3 - replicasCount: 1 # 示例为单 Pod；三节点及以上请改为 3 +```bash +kubectl apply -f cpaas-kafka.yaml +``` - settings: - default_database: observability # 连接 Secret 中需要复用该名称 - merge_tree/materialize_ttl_recalculate_only: "1" - # 自身可观测性系统表会无限增长，最终写满数据盘。 - asynchronous_metric_log/database: system - asynchronous_metric_log/table: asynchronous_metric_log - asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" - metric_log/database: system - metric_log/table: metric_log - metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" - trace_log/database: system - trace_log/table: trace_log - trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" - # 单节点规格使用同 Pod 内嵌 Keeper。三节点及以上请改用下面那份清单， - # 它会为每个 ClickHouse Pod 都运行一个 Keeper。 - keeper_server/tcp_port: "9181" - keeper_server/server_id: "1" - keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log - keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots - keeper_server/coordination_settings/operation_timeout_ms: "10000" - keeper_server/coordination_settings/session_timeout_ms: "30000" - keeper_server/raft_configuration/server/id: "1" - keeper_server/raft_configuration/server/hostname: localhost - keeper_server/raft_configuration/server/port: "9234" +#### 3.2.3 为 broker 的 PVC 预留存储卷（两种规格通用） - zookeeper: - nodes: - - host: localhost - port: 9181 +实例会立即创建自己的 PVC。在为其预留卷之前，这些 PVC 会一直处于 `Pending`：broker 的 PVC 名称里含有按实例生成的 hash，所以卷无法提前准备： - defaults: - templates: - podTemplate: pod-template - dataVolumeClaimTemplate: data-volumeclaim-template - serviceTemplate: service-template +```bash +kubectl -n cpaas-system get pvc \ + -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' +``` - templates: - podTemplates: - - name: pod-template - podDistribution: - - scope: Shard - topologyKey: kubernetes.io/hostname - type: ShardAntiAffinity - spec: - nodeSelector: - node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 - tolerations: - - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 - operator: Exists - effect: NoSchedule - containers: - - name: clickhouse - image: - ports: - - name: http - containerPort: 8123 - - name: client - containerPort: 9000 - - name: interserver - containerPort: 9009 - - name: keeper - containerPort: 9181 - - name: raft - containerPort: 9234 - resources: - requests: - cpu: "1" - memory: 4Gi - limits: - cpu: "2" # 取自规格 - memory: 4Gi # 取自规格 - volumeMounts: - - name: data-volumeclaim-template - mountPath: /var/lib/clickhouse +把每个 PVC 名称原样抄下来（三节点规格为三个 broker、三个 controller；单节点规格为一个 broker、一个 controller），并为每个 PVC 创建一个预绑定的卷。`capacity.storage` 填该 PVC 申请的容量，`local.path` 指向该 Pod 的目录，并用节点亲和把它钉在对应节点上： - serviceTemplates: - - name: service-template - spec: - ports: - - name: http - port: 8123 - - name: tcp - port: 9000 - type: ClusterIP +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-kafka-broker-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-kafka + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-cpaas-kafka-broker--0 + local: + path: /cpaas/data/kafka/broker-0 + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` - volumeClaimTemplates: - - name: data-volumeclaim-template - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 200Gi - storageClassName: # 来自步骤 1 +请把 YAML 保存为 `cpaas-kafka-volumes.yaml`，为全部 PVC 重复 `PersistentVolume` 部分后应用。卷一旦存在，PVC 立即绑定，broker 与 controller 随之启动： + +```bash +kubectl apply -f cpaas-kafka-volumes.yaml ``` -请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： +### 3.3 等待 broker 集群就绪 ```bash -kubectl apply -f cpaas-clickhouse.yaml +kubectl -n cpaas-system get rdsKafka cpaas-kafka \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' +kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka ``` -**三节点及以上。** 请改用下面这份清单，不要用上面那份。Keeper 运行在每一个 ClickHouse Pod 内，Pod 之间自行组成仲裁集群，因此整个实例仍然只是一个 `ClickHouseInstallation`。 +等待 `Ready` 条件变为 `True`，且所有 broker Pod 处于 `Running`。 -静态 Keeper 配置通过 cluster 的 `files` 注入，并用 `include_from` 引入生成出来的文件；与身份相关的部分（`server_id` 和成员列表）由 init 容器按 Pod 生成到内存 `emptyDir` 中。init 容器里的 `SHARDS_COUNT`、`REPLICAS_COUNT` 必须与 `layout.shardsCount`、`layout.replicasCount` 保持一致，否则成员列表不完整，仲裁永远组不起来。 +确认必需配置已下发到 broker： -readiness 探针探测的是 Raft 端口，这是必需的：默认的 HTTP 探针要等 ClickHouse 开始提供服务才会成功，而 ClickHouse 又必须等 Keeper 仲裁集群就绪才能完成启动，于是 Operator 会一直等第一个副本，永远不创建其余副本。 +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" -Keeper 的 `path` 位于 `/var/lib/clickhouse` 之下，也就是挂载的数据卷内，因此 Keeper 的日志与快照和 ClickHouse 数据一样保存在持久卷上。不要把它移到该挂载点之外：放在容器文件系统里的 Keeper 状态会在 Pod 每次重启时丢失。 +kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ + grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes|^log\.retention\.bytes" /tmp/strimzi.properties +``` -`wait-for-self-dns` init 容器会等待 Pod 能解析自己的 headless Service 名称。没有它时，如果某个 Pod 在自己的 DNS 记录发布之前启动，它的分布式 DDL worker 会基于一个解析不了的主机名完成初始化并且不再重试：`CREATE TABLE ... ON CLUSTER` 在其他副本上执行成功，而该副本会静默漏掉这条语句。 +`message.max.bytes` 和 `replica.fetch.max.bytes` 必须是 `10485760`，`log.retention.bytes` 必须与 3.5 中的 topic 上限一致，否则不要继续。broker 侧取值以本文件为准：`kafka-configs.sh --describe` 不会显示这两个静态配置。 + +然后按规格检查节点分布。 + +#### 3.3.1 三节点及以上（生产） + +Kafka Operator 会自动施加硬反亲和，因此三个 broker 必须落在三个不同节点上： + +```bash +kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +``` + +必须出现三个不同的节点名：硬反亲和要求集群至少有 3 个可调度节点。 + +#### 3.3.2 单节点（仅用于验证环境） + +唯一的 broker 和唯一的 controller 共用一个节点，一个可调度节点即可，不需要额外检查。 + +### 3.4 创建 SASL 用户及其 ACL + +日志组件使用同一个账号，需要访问三个 topic、消费组以及 broker 元数据。 ```yaml -apiVersion: clickhouse.altinity.com/v1 -kind: ClickHouseInstallation +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser metadata: - name: cpaas-clickhouse + name: platform-logging namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka spec: - configuration: - users: - # 管理员密码来自上面创建的 Secret - admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password - admin/networks/ip: - - "0.0.0.0/0" - - "::/0" - admin/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION - # 日志组件使用的账号。与 admin 一样在这里声明， - # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 - platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password - platform-logging/networks/ip: - - "0.0.0.0/0" - - "::/0" - platform-logging/profile: default - platform-logging/quota: default - platform-logging/grants/query: - - GRANT ALL ON *.* WITH GRANT OPTION + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + # 三个 topic + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } + # 日志链路使用的消费组 + - host: "*" + operation: All + resource: { type: group, name: alauda_log, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_event, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_audit, patternType: literal } + # LogForward 使用的消费组前缀 + - host: "*" + operation: All + resource: { type: group, name: "logforward-", patternType: prefix } + # 日志查询服务使用的消费组前缀 + - host: "*" + operation: All + resource: { type: group, name: "razor-", patternType: prefix } + # broker 元数据 + - host: "*" + operation: All + resource: { type: cluster, name: kafka-cluster, patternType: literal } +``` - profiles: - default/allow_nondeterministic_mutations: "1" - default/allow_unrestricted_reads_from_keeper: "1" - default/max_execution_time: 120 - default/max_estimated_execution_time: 120 +请把 YAML 保存为 `platform-logging-user.yaml` 并应用： - zookeeper: - nodes: - - host: cpaas-clickhouse-keeper # 3.2 中创建的 Service - port: 9181 +```bash +kubectl apply -f platform-logging-user.yaml +``` - settings: - default_database: observability # 连接 Secret 中需要复用该名称 - merge_tree/materialize_ttl_recalculate_only: "1" - # 自身可观测性系统表会无限增长，最终写满数据盘。 - asynchronous_metric_log/database: system - asynchronous_metric_log/table: asynchronous_metric_log +这九条都必须配置，`operation: All` 已覆盖这些条目所需的读和 describe 权限。请校验： + +```bash +kubectl -n cpaas-system get rdskafkauser platform-logging \ + -o jsonpath='{.status.phase}{"\n"}' # 期望：Active +kubectl -n cpaas-system get secret platform-logging +``` + +消费组名称使用下划线（`alauda_log`），而 topic 名称使用大写字母和下划线（`ALAUDA_LOG_TOPIC`）。 + +### 3.5 创建三个 topic + +资源名必须是合法的 DNS 名称。`spec.topicName` 是 broker 侧名称，必须与上面的 ACL 条目一致。 + +只配置 `retention.ms` 不能限制磁盘占用：一波流量峰值可能在时间窗口到期前就把 broker 卷写满，而卷写满会导致日志、事件、审计三条链路全部停摆，因此必须同时配置 `retention.bytes`。它对 topic 和 broker 都是按分区生效的，取值由 broker 卷容量推导： + +``` +单分区上限 = 卷容量 × 70% ÷（topic 数 × 每 topic 分区数） +``` + +以 200 Gi 卷、3 个 topic、每个 topic 30 分区为例：200 Gi × 70% ÷ 90 = 1,670,265,059 字节，向下取整到完整的 100 MiB segment 后为 `1572864000`。预留的 30% 用于索引文件、`__consumer_offsets`、`__cluster_metadata` 副本，以及永远不会被删除的活跃 segment。三个 topic 的 `retention.bytes` 和 broker 的 `log.retention.bytes` 都要使用同一个值；如果 broker 卷容量不同，请重新计算。 + +#### 3.5.1 三节点及以上（生产） + +下面三个 topic 直接使用 `replicas: 3` 和 `min.insync.replicas: "2"`： + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC # broker 侧名称，必须与 ACL 和连接 Secret 一致 + partitions: 30 # 消费并发度的上限 + replicas: 3 + config: + retention.ms: "172800000" # 48 小时 + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +``` + +请把 YAML 保存为 `alauda-topics.yaml` 并应用： + +```bash +kubectl apply -f alauda-topics.yaml +``` + +确认三个 topic 都已写入容量上限： + +```bash +kubectl -n cpaas-system get rdstopic \ + -o custom-columns='NAME:.metadata.name,TOPIC:.spec.topicName,PARTITIONS:.spec.partitions,RETENTION:.spec.config.retention\.bytes,SEGMENT:.spec.config.segment\.bytes' +``` + +每个 topic 的 `RETENTION` 必须是 `1572864000`，`SEGMENT` 必须是 `104857600`。当 topic 的值与 broker 级 `log.retention.bytes` 相同时，`kafka-configs.sh --describe` 不会显示该值，因此以这里的检查为准。 + +#### 3.5.2 单节点（仅用于验证环境） + +沿用上面三个 topic，把 `replicas` 改为 `1`，`min.insync.replicas` 改为 `"1"`。 + +### 3.6 端到端校验 Kafka 服务 + +先在 broker Pod 内根据用户 Secret 生成客户端配置。broker 监听端口要求 SASL 认证，下面的命令都需要它。 + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ + sh -c 'cat > /tmp/logging-client.properties' <----0`，`` 和 `` 都从 0 开始编号。下面的清单两种规格完全相同，区别只在 PV 数量。使用动态供给时只需创建 StorageClass，跳过 PV。 + +#### 4.1.1 三节点及以上（生产） + +4.4.1 的清单是 1 shard × 3 replica，需要三个 PVC：`data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`、`-0-1-0` 和 `-0-2-0`。下面的清单先预留第一个： + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local-clickhouse + labels: + # 在项目命名空间中是必需的：没有这个授权， + # pvc-validator 准入 webhook 会拒绝所有使用该 StorageClass 的 PVC。 + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-clickhouse-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-clickhouse + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + local: + path: /cpaas/data/clickhouse + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +请把 YAML 保存为 `cpaas-clickhouse-volumes.yaml` 并应用： + +```bash +kubectl apply -f cpaas-clickhouse-volumes.yaml +``` + +请为另外两个副本重复 `PersistentVolume` 部分，`metadata.name` 各不相同，`local.path` 用各自的目录（例如 `/cpaas/data/clickhouse-1`、`/cpaas/data/clickhouse-2`），`values` 填该 Pod 所在节点的 IP。 + +#### 4.1.2 单节点（仅用于验证环境） + +单节点规格只有一个 Pod，只需要一个 PVC：`data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`。上面的清单已经预留了它，直接应用即可，不要再创建额外的 PV。 + +### 4.2 创建密码 Secret + +实例定义了两个账号：用于管理的 `admin`，以及供日志组件使用的 `platform-logging`。两者的密码都来自 Secret，因此请在创建实例之前先把两个 Secret 建好。 + +```bash +kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ + --from-literal=password="$(openssl rand -hex 16)" +kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 4.3 创建 Keeper 客户端 Service + +所有规格都需要 Keeper，单节点也一样，因为日志组件创建的是 `ReplicatedMergeTree` 表；两种规格的区别在于 ClickHouse 如何访问它。 + +#### 4.3.1 三节点及以上（生产） + +三节点及以上时，由 ClickHouse Pod 自身组成 Keeper 仲裁集群：每个 ClickHouse Pod 同时也是一个 Keeper 成员。ClickHouse 通过一个 headless Service 访问该仲裁集群，该 Service 会选中本次安装中所有就绪的 Pod。 + +请把 YAML 保存为 `cpaas-clickhouse-keeper-service.yaml` 并应用： + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + clusterIP: None + type: ClusterIP + ports: + - name: keeper + port: 9181 + protocol: TCP + targetPort: 9181 + selector: + clickhouse.altinity.com/chi: cpaas-clickhouse + clickhouse.altinity.com/namespace: cpaas-system + clickhouse.altinity.com/ready: "yes" + clickhouse.altinity.com/role: keeper +``` + +```bash +kubectl apply -f cpaas-clickhouse-keeper-service.yaml +``` + +其中 `chi`、`namespace`、`ready` 三个标签由 Operator 打上，`role: keeper` 标签来自 4.4.1 的 pod template。 + +#### 4.3.2 单节点（仅用于验证环境） + +单节点规格下，Keeper 通过 4.4.2 中的 `keeper_server/*` 配置运行在唯一的 ClickHouse Pod 内，ClickHouse 通过 `localhost` 访问它。不要创建这个 Service，直接进入 4.4。 + +### 4.4 创建 ClickHouseInstallation + +cluster 名称固定为 `replicated`，需与日志组件保持一致；`shardsCount` 和 `replicasCount` 按步骤 0 的规格设置。下面两份清单请按你选的规格二选一执行。 + +请把 `` 替换为平台 middleware 包发布的 ClickHouse server 镜像，例如 `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`。 + +#### 4.4.1 三节点及以上（生产） + +生产环境请使用这份清单。Keeper 运行在每一个 ClickHouse Pod 内，Pod 之间自行组成仲裁集群，因此整个实例仍然只是一个 `ClickHouseInstallation`。 + +静态 Keeper 配置通过 cluster 的 `files` 注入，并用 `include_from` 引入生成出来的文件；与身份相关的部分（`server_id` 和成员列表）由 init 容器按 Pod 生成到内存 `emptyDir` 中。init 容器里的 `SHARDS_COUNT`、`REPLICAS_COUNT` 必须与 `layout.shardsCount`、`layout.replicasCount` 保持一致，否则成员列表不完整，仲裁永远组不起来。 + +readiness 探针探测的是 Raft 端口，这是必需的：默认的 HTTP 探针要等 ClickHouse 开始提供服务才会成功，而 ClickHouse 又必须等 Keeper 仲裁集群就绪才能完成启动，于是 Operator 会一直等第一个副本，永远不创建其余副本。 + +Keeper 的 `path` 位于 `/var/lib/clickhouse` 之下，也就是挂载的数据卷内，因此 Keeper 的日志与快照和 ClickHouse 数据一样保存在持久卷上。不要把它移到该挂载点之外：放在容器文件系统里的 Keeper 状态会在 Pod 每次重启时丢失。 + +`wait-for-self-dns` init 容器会等待 Pod 能解析自己的 headless Service 名称。没有它时，如果某个 Pod 在自己的 DNS 记录发布之前启动，它的分布式 DDL worker 会基于一个解析不了的主机名完成初始化并且不再重试：`CREATE TABLE ... ON CLUSTER` 在其他副本上执行成功，而该副本会静默漏掉这条语句。 + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # 管理员密码来自上面创建的 Secret + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # 日志组件使用的账号。与 admin 一样在这里声明， + # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper # 4.3.1 中创建的 Service + port: 9181 + + settings: + default_database: observability # 连接 Secret 中需要复用该名称 + merge_tree/materialize_ttl_recalculate_only: "1" + # 自身可观测性系统表会无限增长，最终写满数据盘。 + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" metric_log/database: system metric_log/table: metric_log @@ -501,7 +928,7 @@ spec: type: ShardAntiAffinity metadata: labels: - clickhouse.altinity.com/role: keeper # 由 3.2 的 Service 选中 + clickhouse.altinity.com/role: keeper # 由 4.3.1 的 Service 选中 spec: nodeSelector: node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 @@ -623,484 +1050,284 @@ spec: emptyDir: medium: Memory serviceTemplates: - - name: service-template - spec: - ports: - - name: http - port: 8123 - - name: tcp - port: 9000 - type: ClusterIP - volumeClaimTemplates: - - name: data-volumeclaim-template - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 200Gi - storageClassName: # 来自步骤 1 -``` - -请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： - -```bash -kubectl apply -f cpaas-clickhouse.yaml -``` - -`default_database: observability` 会让 ClickHouse 在启动时创建 `observability` 库，这里不需要再建库。 - -### 3.4 等待集群就绪 - -```bash -kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ - -o jsonpath='{.status.status}{"\n"}' # 反复执行，直到变为：Completed - -kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse -kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse -kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse -``` - -所有 ClickHouse Pod 必须处于 `Running` 且就绪，且每个 PVC 都必须是 `Bound`。StorageClass 不存在或无法绑定时不会有任何 Pod，也不会有报错，因此不能只看 `ClickHouseInstallation` 的 status。 - -三节点及以上请先确认 Keeper 仲裁集群再继续。Keeper 运行在 ClickHouse Pod 内，因此逐个副本检查一个 Pod： - -```bash -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state -``` - -三条命令的结果必须是 1 个 `leader`、2 个 `follower`。同时确认 ClickHouse 是通过 3.2 的 Service 访问仲裁集群的： - -```bash -kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ - clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" -# 期望：cpaas-clickhouse-keeper 9181 -``` - -### 3.5 读取日志账号密码 - -该账号已在 3.3 应用的 `ClickHouseInstallation` 中声明，因此这里不需要创建账号，也没有 `GRANT` 语句要执行：它和 `admin` 一样是服务端配置用户，自带日志组件所需的权限。 - -请读取它的密码，供步骤 6 记录连接信息。3.6 会复用 `CH_POD` 与 `LOG_PASSWORD` 变量，因此两步请在同一个 shell 中执行。 - -```bash -CH_POD="$(kubectl -n cpaas-system get pod \ - -l clickhouse.altinity.com/chi=cpaas-clickhouse \ - -o jsonpath='{.items[0].metadata.name}')" -LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ - -o jsonpath='{.data.password}' | base64 -d)" -echo "platform-logging password: $LOG_PASSWORD" + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # 来自 4.1 ``` -### 3.6 校验 - -请在同一个 shell 中执行（3.5 定义的 `$CH_POD` 和 `$LOG_PASSWORD` 还在）： +请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： ```bash -kubectl -n cpaas-system exec "$CH_POD" -- \ - clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ - --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" - -kubectl -n cpaas-system exec "$CH_POD" -- \ - clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ - --query "DROP TABLE observability.__perm_check" +kubectl apply -f cpaas-clickhouse.yaml ``` -两条命令都必须成功，建表失败说明该账号无法管理表结构。 - -记录以下信息： - -| 值 | 从哪里获取 | -| --- | --- | -| 连接地址 | 暴露 `8123` 的集群 Service，可用 `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse` 查看 | -| Cluster 名称 | `spec.configuration.clusters[].name` | -| 数据库 | `spec.configuration.settings.default_database` | -| 分片数 / 副本数 | `shardsCount` / `replicasCount` | -| 用户名 / 密码 | 上面创建的账号 | - -## 步骤 4：创建 Kafka 服务 - -### 4.1 创建 SASL 密码 Secret - -在 Alauda OS 节点或其他启用了 FIPS 的主机上，密码长度必须不少于 32 个字符。 +`default_database: observability` 会让 ClickHouse 在启动时创建 `observability` 库，这里不需要再建库。 -```bash -kubectl -n cpaas-system create secret generic platform-logging-password \ - --from-literal=password="$(openssl rand -hex 16)" -``` +#### 4.4.2 单节点（仅用于验证环境） -### 4.2 创建 broker 集群 +Keeper 通过 `keeper_server/*` 配置运行在 ClickHouse Pod 内。 ```yaml -apiVersion: middleware.alauda.io/v1 -kind: RdsKafka +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation metadata: - name: cpaas-kafka + name: cpaas-clickhouse namespace: cpaas-system spec: - mode: KRaft - version: 4.2.0 # Alauda Kafka Operator 支持的最低版本 - replicas: 3 - resources: - limits: { cpu: "2", memory: 4Gi } # 取自规格 - requests: { cpu: 500m, memory: 2Gi } - storage: - size: 200Gi - class: - deleteClaim: false - controller: - replicas: 3 - roles: ["controller"] # 必需：缺少时 node pool 会被拒绝 - template: - pod: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 - operator: Exists - tolerations: - - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 - operator: Exists - effect: NoSchedule - resources: - limits: { cpu: "1", memory: 2Gi } - requests: { cpu: 100m, memory: 512Mi } - storage: - size: 20Gi - class: - deleteClaim: false - kafka: - template: - pod: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 - operator: Exists - tolerations: - - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 - operator: Exists - effect: NoSchedule - listeners: - plain: - authentication: - type: scram-sha-512 - tls: - authentication: - type: scram-sha-512 - authorization: - type: simple - config: - auto.create.topics.enable: "false" - default.replication.factor: "3" - min.insync.replicas: "2" - offsets.topic.replication.factor: "3" - transaction.state.log.replication.factor: "3" - transaction.state.log.min.isr: "2" - log.retention.hours: "48" - unclean.leader.election.enable: "false" - message.max.bytes: "10485760" - replica.fetch.max.bytes: "10485760" - socket.request.max.bytes: "104857600" - entityOperator: - topicOperator: {} # 必需：由它创建 4.5 中的 topic - userOperator: {} # 必需：由它创建 4.4 中的 SASL 用户 -``` - -请把 YAML 保存为 `cpaas-kafka.yaml` 并应用： - -```bash -kubectl apply -f cpaas-kafka.yaml -``` - -实例会立即创建自己的 PVC。在为其预留卷之前，这些 PVC 会一直处于 `Pending`：broker 的 PVC 名称里含有按实例生成的 hash，所以卷无法提前准备： - -```bash -kubectl -n cpaas-system get pvc \ - -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' -``` + configuration: + users: + # 管理员密码来自上面创建的 Secret + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # 日志组件使用的账号。与 admin 一样在这里声明， + # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION -把这六个名称原样抄下来（三个 `...-broker-...`、三个 `...-controller-...`），并为每个 PVC 创建一个预绑定的卷。`capacity.storage` 填该 PVC 申请的容量，`local.path` 指向该 Pod 的目录，并用节点亲和把它钉在对应节点上： + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 -```yaml -apiVersion: v1 -kind: PersistentVolume -metadata: - name: cpaas-kafka-broker-0 -spec: - capacity: - storage: 200Gi - volumeMode: Filesystem - accessModes: - - ReadWriteOnce - persistentVolumeReclaimPolicy: Retain - storageClassName: cpaas-local-kafka - claimRef: # 该卷只留给下面这个 PVC - apiVersion: v1 - kind: PersistentVolumeClaim - namespace: cpaas-system - name: data-cpaas-kafka-broker--0 - local: - path: /cpaas/data/kafka/broker-0 - nodeAffinity: - required: - nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/hostname - operator: In - values: [""] -``` + clusters: + - name: replicated # 连接 Secret 中需要复用该名称 + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # 取自规格：1、1、2 或 3 + replicasCount: 1 # 示例为单 Pod；三节点及以上请改为 3 -请把 YAML 保存为 `cpaas-kafka-volumes.yaml`，为全部六个 PVC 重复 `PersistentVolume` 部分后应用。卷一旦存在，PVC 立即绑定，broker 与 controller 随之启动： + settings: + default_database: observability # 连接 Secret 中需要复用该名称 + merge_tree/materialize_ttl_recalculate_only: "1" + # 自身可观测性系统表会无限增长，最终写满数据盘。 + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + # 单节点规格使用同 Pod 内嵌 Keeper。三节点及以上请改用下面那份清单， + # 它会为每个 ClickHouse Pod 都运行一个 Keeper。 + keeper_server/tcp_port: "9181" + keeper_server/server_id: "1" + keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log + keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots + keeper_server/coordination_settings/operation_timeout_ms: "10000" + keeper_server/coordination_settings/session_timeout_ms: "30000" + keeper_server/raft_configuration/server/id: "1" + keeper_server/raft_configuration/server/hostname: localhost + keeper_server/raft_configuration/server/port: "9234" -```bash -kubectl apply -f cpaas-kafka-volumes.yaml -``` + zookeeper: + nodes: + - host: localhost + port: 9181 -以下配置不能省略： + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template -| 配置项 | 为什么必须设置 | -| --- | --- | -| `message.max.bytes: "10485760"` | 审计数据的批大小约为 1.1–1.5 MiB，Kafka 默认的 1 MiB 会拒绝所有审计批次。 | -| `replica.fetch.max.bytes: "10485760"` | 必须不小于 `message.max.bytes`，否则副本同步会停滞。 | -| `auto.create.topics.enable: "false"` | 避免拼错的 topic 名称被自动创建并静默接收数据。 | -| `entityOperator.topicOperator` / `userOperator` | 缺少它们时，下一步中的 `RdsTopic` 和 `RdsKafkaUser` 不会生效到 broker。 | + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: keeper + containerPort: 9181 + - name: raft + containerPort: 9234 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse -### 4.3 等待 broker 集群就绪 + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP -```bash -kubectl -n cpaas-system get rdsKafka cpaas-kafka \ - -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' -kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # 来自 4.1 ``` -等待 `Ready` 条件变为 `True`，且所有 broker Pod 处于 `Running`。Kafka Operator 会自动施加硬反亲和，因此三个 broker 必须落在三个不同节点上： +请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： ```bash -kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ - -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +kubectl apply -f cpaas-clickhouse.yaml ``` -必须出现三个不同的节点名：硬反亲和要求集群至少有 3 个可调度节点。 - -确认必需配置已下发到 broker： +### 4.5 等待集群就绪 ```bash -KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ - -o jsonpath='{.items[0].metadata.name}')" +kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ + -o jsonpath='{.status.status}{"\n"}' # 反复执行，直到变为：Completed -kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ - grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes" /tmp/strimzi.properties +kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse ``` -两个值都必须是 `10485760`，否则不要继续。这些配置以静态配置形式写入 broker，`kafka-configs.sh --describe` 看不到它们。 +所有 ClickHouse Pod 必须处于 `Running` 且就绪，且每个 PVC 都必须是 `Bound`。StorageClass 不存在或无法绑定时不会有任何 Pod，也不会有报错，因此不能只看 `ClickHouseInstallation` 的 status。 -### 4.4 创建 SASL 用户及其 ACL +#### 4.5.1 三节点及以上（生产） -日志组件使用同一个账号，需要访问三个 topic、消费组以及 broker 元数据。 +请先确认 Keeper 仲裁集群再继续。Keeper 运行在 ClickHouse Pod 内，因此逐个副本检查一个 Pod： -```yaml -apiVersion: middleware.alauda.io/v1 -kind: RdsKafkaUser -metadata: - name: platform-logging - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - authentication: - type: scram-sha-512 - password: - valueFrom: - secretKeyRef: - name: platform-logging-password - key: password - authorization: - type: simple - acls: - # 三个 topic - - host: "*" - operation: All - resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } - - host: "*" - operation: All - resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } - - host: "*" - operation: All - resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } - # 日志链路使用的消费组 - - host: "*" - operation: All - resource: { type: group, name: alauda_log, patternType: literal } - - host: "*" - operation: All - resource: { type: group, name: alauda_event, patternType: literal } - - host: "*" - operation: All - resource: { type: group, name: alauda_audit, patternType: literal } - # LogForward 使用的消费组前缀 - - host: "*" - operation: All - resource: { type: group, name: "logforward-", patternType: prefix } - # 日志查询服务使用的消费组前缀 - - host: "*" - operation: All - resource: { type: group, name: "razor-", patternType: prefix } - # broker 元数据 - - host: "*" - operation: All - resource: { type: cluster, name: kafka-cluster, patternType: literal } +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state ``` -请把 YAML 保存为 `platform-logging-user.yaml` 并应用： +三条命令的结果必须是 1 个 `leader`、2 个 `follower`。同时确认 ClickHouse 是通过 4.3.1 的 Service 访问仲裁集群的： ```bash -kubectl apply -f platform-logging-user.yaml +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# 期望：cpaas-clickhouse-keeper 9181 ``` -这九条都必须配置，`operation: All` 已覆盖这些条目所需的读和 describe 权限。请校验： +#### 4.5.2 单节点（仅用于验证环境） + +单节点只有一个 Keeper 成员，状态是 `standalone`，不存在 leader 与 follower。确认它的状态和监听地址： ```bash -kubectl -n cpaas-system get rdskafkauser platform-logging \ - -o jsonpath='{.status.phase}{"\n"}' # 期望：Active -kubectl -n cpaas-system get secret platform-logging -``` +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +# 期望：standalone -消费组名称使用下划线（`alauda_log`），而 topic 名称使用大写字母和下划线（`ALAUDA_LOG_TOPIC`）。 +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# 期望：localhost 9181 +``` -### 4.5 创建三个 topic +### 4.6 读取日志账号密码 -```yaml -apiVersion: middleware.alauda.io/v1 -kind: RdsTopic -metadata: - name: alauda-log-topic - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - topicName: ALAUDA_LOG_TOPIC # broker 侧名称，必须与 ACL 和连接 Secret 一致 - partitions: 30 # 消费并发度的上限 - replicas: 3 - config: - retention.ms: "172800000" # 48 小时 - segment.bytes: "1073741824" - min.insync.replicas: "2" - compression.type: producer - max.message.bytes: "10485760" ---- -apiVersion: middleware.alauda.io/v1 -kind: RdsTopic -metadata: - name: alauda-event-topic - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - topicName: ALAUDA_EVENT_TOPIC - partitions: 30 - replicas: 3 - config: - retention.ms: "172800000" - segment.bytes: "1073741824" - min.insync.replicas: "2" - compression.type: producer - max.message.bytes: "10485760" ---- -apiVersion: middleware.alauda.io/v1 -kind: RdsTopic -metadata: - name: alauda-audit-topic - namespace: cpaas-system - labels: - middleware.alauda.io/cluster: cpaas-kafka -spec: - topicName: ALAUDA_AUDIT_TOPIC - partitions: 30 - replicas: 3 - config: - retention.ms: "172800000" - segment.bytes: "1073741824" - min.insync.replicas: "2" - compression.type: producer - max.message.bytes: "10485760" -``` +该账号已在 4.4 应用的 `ClickHouseInstallation` 中声明，因此这里不需要创建账号，也没有 `GRANT` 语句要执行：它和 `admin` 一样是服务端配置用户，自带日志组件所需的权限。 -请把 YAML 保存为 `alauda-topics.yaml` 并应用： +请读取它的密码，供步骤 6 记录连接信息。4.7 会复用 `CH_POD` 与 `LOG_PASSWORD` 变量，因此两步请在同一个 shell 中执行。 ```bash -kubectl apply -f alauda-topics.yaml +CH_POD="$(kubectl -n cpaas-system get pod \ + -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o jsonpath='{.items[0].metadata.name}')" +LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "platform-logging password: $LOG_PASSWORD" ``` -资源名必须是合法的 DNS 名称。`spec.topicName` 是 broker 侧名称，必须与上面的 ACL 条目一致。 - -示例中的保留时间为 48 小时。只按时间保留并不能限制磁盘占用：一波流量峰值可能在保留窗口到期前就把 broker 卷写满。请按「保留窗口内的峰值速率」规划 broker 卷容量，或者给三个 topic 都加上 `retention.bytes`。`retention.bytes` 是按分区生效的，因此 broker 卷需要能容纳 `分区数 × retention.bytes`。 +### 4.7 校验 -### 4.6 端到端校验 Kafka 服务 - -先在 broker Pod 内根据用户 Secret 生成客户端配置。broker 监听端口要求 SASL 认证，下面的命令都需要它。 +请在同一个 shell 中执行（4.6 定义的 `$CH_POD` 和 `$LOG_PASSWORD` 还在）： ```bash -KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ - -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ - -o jsonpath='{.items[0].metadata.name}')" +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" -kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ - sh -c 'cat > /tmp/logging-client.properties' <-kafka-bootstrap.cpaas-system.svc:9093`，不使用 TLS 的 SASL 为 `:9092` | -| Cluster 名称 | `RdsKafka` 资源的 `metadata.name` | -| 用户名 / 密码 | `RdsKafkaUser` 的名称及其密码 | -| Topic | `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC`、`ALAUDA_AUDIT_TOPIC` | -| CA 证书 | 仅在用 `9093` 的 TLS 监听端口时需要。Kafka Operator 会把它放在 `cpaas-system` 下名称以 `-cluster-ca-cert` 结尾的 Secret 中 | +1. 在每台将运行 OpenSearch Pod 的节点上创建数据目录并设置属主（uid 1000）。在 Alauda OS 节点上请把 `/cpaas` 换成 `/var/cpaas`： + + ```bash + sudo mkdir -p /cpaas/data/opensearch + sudo chown -R 1000:1000 /cpaas/data/opensearch + ``` -## 步骤 5：创建 OpenSearch 集群 +2. 确认 `vm.max_map_count` 不小于 `262144`。OpenSearch Operator 会通过 init 容器设置它，因此通常不用处理；只有集群启用了受限的 Pod Security Admission 时，该 init 容器无法设置，才需要在每个节点上手工设置： -目标存储为 ClickHouse 时请跳过本步骤。 + ```bash + sudo sysctl -w vm.max_map_count=262144 + echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf + ``` -### 5.1 创建集群 +3. 使用静态本地卷时，需要按步骤 1 的模板为每个节点预先创建预绑定 PV，`claimRef` 填 `data---`。对应下面示例就是 `data-cpaas-opensearch-masters-0` 到 `data-cpaas-opensearch-masters-2`，以及 `data-cpaas-opensearch-data-0` 到 `data-cpaas-opensearch-data-4`。使用动态供给时跳过本项。 -按步骤 0 的规格设置节点池。示例为 3 + 5：三个 master、五个 data。小规格只用一个节点池，`roles` 设为 `[cluster_manager, data]`，`replicas` 设为 3 或 5。 +请把 YAML 保存为 `cpaas-opensearch.yaml` 并应用： ```yaml apiVersion: opensearch.opster.io/v1 @@ -1172,7 +1399,7 @@ spec: replicas: 0 ``` -请把 YAML 保存为 `cpaas-opensearch.yaml` 并应用： +应用： ```bash kubectl apply -f cpaas-opensearch.yaml @@ -1305,44 +1532,74 @@ curl -sk -u "admin:" -X DELETE "$OS/_index_template/perm-check" curl -sk -u "admin:" -X DELETE "$OS/log-perm-check" ``` -记录以下信息： - -| 值 | 从哪里获取 | -| --- | --- | -| 连接地址 | Service 地址或负载均衡地址，例如 `https://:9200` | -| 用户名 / 密码 | 上面创建的账号 | -| CA 证书 | 仅在连接地址使用私有 CA 时需要。Operator 会把它创建在 `cpaas-system` 下的 Secret `cpaas-opensearch-ca` 中 | +连接信息请在步骤 6 记录。 平台下发的索引模板为 1 分片 1 副本。如果你的高可用策略需要不同的取值，请下发一个优先级更高的可组合模板，并用 `GET /_index_template` 确认结果。已创建的索引会保持创建时的设置。 ## 步骤 6：记录连接信息 -请记录下表各项。 +请记录下表各项。中间一列是安装时连接 Secret 中对应的 key。 + +### ClickHouse + +| 值 | 连接 Secret key | 从哪里获取 | +| --- | --- | --- | +| 连接地址 | `platform-default-ch-conn` → `endpoint` | 暴露 8123 端口的 cluster Service，可用 `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse` 查询。启用 TLS 时用 `https://` 和 8443 端口 | +| Cluster 名称 | `cluster` | `ClickHouseInstallation` 的 `spec.configuration.clusters[0].name`，也是 `ON CLUSTER` 使用的名称。默认值：`replicated` | +| 数据库 | `database` | `spec.configuration.settings.default_database`。默认值：`observability` | +| 分片数 / 副本数 | 不在 Secret 中：`PlatformLogForward` 的 `externalStorage.shards` / `replicas` | `spec.configuration.clusters[0].layout.shardsCount` / `replicasCount` | +| 用户名 | `username` | `ClickHouseInstallation` 中声明的日志账号。默认值：`platform-logging` | +| 密码 | `password` | `kubectl -n cpaas-system get secret clickhouse-platform-logging-password -o jsonpath='{.data.password}' \| base64 -d` | +| CA 证书 | `tls.ca` | 仅当连接地址使用 HTTPS 且为自签 CA 时需要 | + +### OpenSearch + +| 值 | 连接 Secret key | 从哪里获取 | +| --- | --- | --- | +| 连接地址 | `platform-default-os-conn` → `endpoints` | 服务地址或负载均衡地址，例如 `https://:9200` 或 `https://cpaas-opensearch.cpaas-system.svc:9200`。请把高可用地址放在第一个 | +| 用户名 | `username` | 5.3 中创建的账号。默认值：`platform-logging` | +| 密码 | `password` | 你为该账号设置的密码 | -| 值 | ClickHouse | OpenSearch | Kafka | -| --- | --- | --- | --- | -| 连接地址 | 必需 | 必需 | 必需 | -| Cluster 名称 | 必需 | — | 必需 | -| 数据库 | 必需 | — | — | -| 分片数 / 副本数 | 必需 | — | — | -| 用户名 | 必需 | 必需 | 必需 | -| 密码 | 必需 | 必需 | 必需 | -| Topic | — | — | 必需 | -| CA 证书 | 连接地址使用私有 CA 时必需 | 连接地址使用私有 CA 时必需（用于历史数据迁移） | 连接地址使用私有 CA 时必需 | +### Kafka + +| 值 | 连接 Secret key | 从哪里获取 | +| --- | --- | --- | +| Bootstrap 地址 | `platform-default-mq-conn` → `bootstrap` | `:9093` 走 SASL over TLS 时为 `-kafka-bootstrap.cpaas-system.svc:9093`；不走 TLS 时用 `:9092` | +| Cluster 名称 | `kafkaClusterName` | `RdsKafka` 资源的 `metadata.name`。默认值：`cpaas-kafka` | +| 用户名 | `username` | `RdsKafkaUser` 的名称。默认值：`platform-logging` | +| 密码 | `password` | `kubectl -n cpaas-system get secret platform-logging-password -o jsonpath='{.data.password}' \| base64 -d`。在 Alauda OS 节点或其他启用 FIPS 的主机上必须至少 32 个字符 | +| Topic | `topics.log` / `topics.event` / `topics.audit` | `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC`、`ALAUDA_AUDIT_TOPIC` | +| SASL 机制 | `sasl_mechanism` | 除 broker 使用其他机制外，固定为 `SCRAM-SHA-512` | +| CA 证书 | `tls.ca` | 仅 9093 的 TLS 监听器需要：`kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' \| base64 -d` | ## 环境检查清单 +### 通用 + | 检查项 | 期望结果 | | --- | --- | -| 节点和磁盘 | 已使用独占节点并打好标签与污点，已挂载 SSD，目录已按正确的属主创建，且清单中已容忍该污点 | -| StorageClass | 已存在，并且能够为存储集群绑定存储卷 | -| Operator | ClickHouse、Kafka 以及（目标为 OpenSearch 时）OpenSearch 的 Operator 已就绪，其 CRD 已存在，且都能 watch `cpaas-system` | -| ClickHouse | `status.status` 为 `Completed`，所有 Pod 就绪，日志账号可以建表和删表；三节点及以上时 Keeper 仲裁集群为 1 个 leader、2 个 follower | -| OpenSearch | 集群健康状态为 `green`，日志账号可以管理索引模板；启用插件时，每个节点都能看到 `analysis-ik` 且能对中文分词 | -| Kafka broker | 集群就绪，`message.max.bytes` 和 `replica.fetch.max.bytes` 均为 `10485760` | +| 节点和磁盘 | 已使用独占节点并打好标签与污点，已挂载用于持久化的 SSD，且清单中已容忍该污点 | +| StorageClass | 每个要安装的组件都有对应的 StorageClass，并能正常绑定存储卷 | +| Operator | `log-storage-operator`、ClickHouse 或 OpenSearch 对应的 Operator、Kafka Operator 均已就绪，其 CRD 已存在，且都能 watch `cpaas-system` | +| Kafka broker | 集群就绪，`message.max.bytes` 和 `replica.fetch.max.bytes` 均为 `10485760`，且 `log.retention.bytes` 与 topic 上限一致 | | Kafka 用户和 ACL | `RdsKafkaUser` 为 `Active`，九条 ACL 均已配置 | -| Kafka topic | broker 上已存在三个 topic，且分区数和保留时间符合预期 | +| Kafka topic | broker 上已存在三个 topic，且分区数、`retention.bytes` 和 `segment.bytes` 符合预期 | | Kafka 连通性 | 已使用日志账号成功生产和消费一条记录 | -| 信息记录 | 连接地址、cluster、数据库、拓扑、凭据、topic 和 CA 证书均已记录 | + +### ClickHouse + +| 检查项 | 期望结果 | +| --- | --- | +| ClickHouse | `status.status` 为 `Completed`，所有 Pod 就绪，日志账号可以建表和删表。三节点及以上：Keeper 仲裁集群为 1 个 leader、2 个 follower。单节点：唯一的 Keeper 成员状态为 `standalone` | +| Keeper Service | 三节点及以上：4.3.1 创建的 Headless Service 能选中就绪的 Pod，且 `system.zookeeper_connection` 指向它。单节点：不创建 Service，`system.zookeeper_connection` 指向 `localhost` | +| 信息记录 | 连接地址、cluster 名称、数据库、分片数 / 副本数、用户名、密码，以及使用 HTTPS 时的 CA 证书均已记录 | + +### OpenSearch + +| 检查项 | 期望结果 | +| --- | --- | +| OpenSearch | 集群健康状态为 `green`，且日志账号可以管理索引模板 | +| 中文分词 | 启用插件时，每个节点都能看到 `analysis-ik` 且能对中文分词 | +| 信息记录 | 连接地址、用户名和密码均已记录 | 任何一项不通过，都必须在日志组件连接该存储之前修复。 From bcc226216d9510c0a9b95796baedb92be4bbc79c Mon Sep 17 00:00:00 2001 From: root Date: Fri, 18 Sep 2026 14:06:08 +0800 Subject: [PATCH 38/38] docs: guard the fresh install against an auto-generated Adopt PLF The logging operator's auto-migration treats ClickHouseInstallation/cpaas-clickhouse, RdsKafka/cpaas-kafka, and Secret/clickhouse-basic-auth as a chart-installed deployment. Those are the names Environment Preparation creates, so an operator start after preparation makes it create PlatformLogForward/platform-default with installMode=Adopt. installMode is immutable, so the Installation manifest is then rejected with "installMode is immutable". - install: check for an existing PlatformLogForward before applying, and document the delete-and-apply recovery plus the connection Secret ownership annotation - prepare: install the operators before creating the storage resources --- docs/en/install/index.mdx | 44 +++++++++++++++++++++++++++++++++++++-- docs/en/prepare/index.mdx | 1 + docs/zh/install/index.mdx | 44 +++++++++++++++++++++++++++++++++++++-- docs/zh/prepare/index.mdx | 1 + 4 files changed, 86 insertions(+), 4 deletions(-) diff --git a/docs/en/install/index.mdx b/docs/en/install/index.mdx index e5f6e37..65877e0 100644 --- a/docs/en/install/index.mdx +++ b/docs/en/install/index.mdx @@ -175,12 +175,32 @@ A wrong value in a multi-shard or replicated deployment leaves part of the Click `PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. These are the replica counts of the logging components themselves, not of ClickHouse: a single-node ClickHouse still runs them as they are. -Save the YAML as `platform-log-forward.yaml` and apply it: +Save the YAML as `platform-log-forward.yaml`, then check whether the logging operator already created this resource on its own: + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +The operator inspects the namespace once at each start. When it finds `ClickHouseInstallation/cpaas-clickhouse`, `RdsKafka/cpaas-kafka`, or `Secret/clickhouse-basic-auth` it assumes an older chart installed them and creates `PlatformLogForward/platform-default` itself with `installMode: Adopt`. `installMode` is immutable, so applying this manifest against that object fails with `installMode is immutable`. + +When the first command prints `Adopt` and `AppRelease/logclickhouse` does not exist, nothing was adopted from a chart. Delete the object and apply this manifest in one step, then mark the two connection Secrets as yours so the operator stops deriving them: + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-ch-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +Otherwise apply the manifest: ```bash kubectl apply -f platform-log-forward.yaml ``` +When `AppRelease/logclickhouse` exists, stop and use the migration chapter instead: that cluster holds chart-installed logging data. + ### 2.3 Verify Watch the status until it finishes, and press `Ctrl+C` to stop: @@ -271,12 +291,32 @@ spec: `PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. -Save the YAML as `platform-log-forward.yaml` and apply it: +Save the YAML as `platform-log-forward.yaml`, then check whether the logging operator already created this resource on its own: + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +The operator inspects the namespace once at each start. When it finds `ClickHouseInstallation/cpaas-clickhouse`, `RdsKafka/cpaas-kafka`, or `Secret/clickhouse-basic-auth` it assumes an older chart installed them and creates `PlatformLogForward/platform-default` itself with `installMode: Adopt`. `installMode` is immutable, so applying this manifest against that object fails with `installMode is immutable`. + +When the first command prints `Adopt` and `AppRelease/logclickhouse` does not exist, nothing was adopted from a chart. Delete the object and apply this manifest in one step, then mark the connection Secrets as yours so the operator stops deriving them: + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-os-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +Otherwise apply the manifest: ```bash kubectl apply -f platform-log-forward.yaml ``` +When `AppRelease/logclickhouse` exists, stop and use the migration chapter instead: that cluster holds chart-installed logging data. + ### 3.3 Verify Watch the status until it finishes, and press `Ctrl+C` to stop: diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx index 8e993ef..8cf9a47 100644 --- a/docs/en/prepare/index.mdx +++ b/docs/en/prepare/index.mdx @@ -134,6 +134,7 @@ Install the operators from the platform marketplace. The `ClickHouseInstallation | Alauda Kafka operator (`strimzi-kafka-operator`) | `kafka-system` | All namespaces | | `opensearch-operator` (OpenSearch only) | `opensearch-operator` | All namespaces | +- Install these operators before you create the resources in Steps 3 to 5. `log-storage-operator` inspects the namespace once at each start: it treats a `ClickHouseInstallation` named `cpaas-clickhouse`, an `RdsKafka` named `cpaas-kafka`, or a `Secret` named `clickhouse-basic-auth` as a deployment installed by an older chart. - Install only the operators that are missing. If one is already installed on the cluster, for example by an earlier release, keep it and do not install a second copy: two copies of the same operator write to the same `cpaas-system` resources. Check its watch scope instead and widen it if needed. - Do not create an OperatorGroup in `cpaas-system`. The platform already owns one there, and a second OperatorGroup makes the platform reject every Subscription in that namespace, including its own. - For `kafka-system` and `opensearch-operator`, the OperatorGroup must have no `spec.targetNamespaces`. If an OperatorGroup scoped to its own namespace already exists, remove the field and wait for the operator pod to restart. diff --git a/docs/zh/install/index.mdx b/docs/zh/install/index.mdx index 2deacb7..c9bbf7d 100644 --- a/docs/zh/install/index.mdx +++ b/docs/zh/install/index.mdx @@ -173,12 +173,32 @@ spec: `PlatformLogForward` 是集群级资源，不要添加 `metadata.namespace`；`secretRef` 内的 `namespace` 仍用于定位 `cpaas-system` 中的连接 Secret。CRD 默认 `aggregateVector.replicas: 3`、`razor.replicas: 2`，如果容量或部署规划要求不同副本数，请显式设置。这两个是日志组件自身的副本数，与 ClickHouse 规格无关：单节点 ClickHouse 也照常使用它们。 -将 YAML 保存为 `platform-log-forward.yaml` 并应用： +将 YAML 保存为 `platform-log-forward.yaml`，然后确认日志 Operator 是否已经自动创建了这个资源： + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +Operator 每次启动都会检查一次命名空间。只要发现 `ClickHouseInstallation/cpaas-clickhouse`、`RdsKafka/cpaas-kafka` 或 `Secret/clickhouse-basic-auth`，它就认为这是更早版本的 chart 装的，并自行创建 `PlatformLogForward/platform-default`，且 `installMode` 为 `Adopt`。`installMode` 不可修改，此时应用本文的清单会报 `installMode is immutable`。 + +如果第一条命令输出 `Adopt`，且 `AppRelease/logclickhouse` 不存在，说明并没有从 chart 接管任何数据。请先删除该对象、再应用本文清单（一条命令内完成），然后把这几个连接 Secret 标记为自己的，避免 Operator 再次派生： + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-ch-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +资源不存在时，正常应用清单： ```bash kubectl apply -f platform-log-forward.yaml ``` +如果 `AppRelease/logclickhouse` 存在，请停止操作并改用迁移章节：该集群上已有 chart 安装的日志数据。 + ### 2.3 校验 观察状态直到完成，按 `Ctrl+C` 退出： @@ -267,12 +287,32 @@ spec: `PlatformLogForward` 是集群级资源，不要添加 `metadata.namespace`；`secretRef` 内的 `namespace` 仍用于定位 `cpaas-system` 中的连接 Secret。CRD 默认 `aggregateVector.replicas: 3`、`razor.replicas: 2`，如果容量或部署规划要求不同副本数，请显式设置。 -将 YAML 保存为 `platform-log-forward.yaml` 并应用： +将 YAML 保存为 `platform-log-forward.yaml`，然后确认日志 Operator 是否已经自动创建了这个资源： + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +Operator 每次启动都会检查一次命名空间。只要发现 `ClickHouseInstallation/cpaas-clickhouse`、`RdsKafka/cpaas-kafka` 或 `Secret/clickhouse-basic-auth`，它就认为这是更早版本的 chart 装的，并自行创建 `PlatformLogForward/platform-default`，且 `installMode` 为 `Adopt`。`installMode` 不可修改，此时应用本文的清单会报 `installMode is immutable`。 + +如果第一条命令输出 `Adopt`，且 `AppRelease/logclickhouse` 不存在，说明并没有从 chart 接管任何数据。请先删除该对象、再应用本文清单（一条命令内完成），然后把这几个连接 Secret 标记为自己的，避免 Operator 再次派生： + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-os-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +资源不存在时，正常应用清单： ```bash kubectl apply -f platform-log-forward.yaml ``` +如果 `AppRelease/logclickhouse` 存在，请停止操作并改用迁移章节：该集群上已有 chart 安装的日志数据。 + ### 3.3 校验 观察状态直到完成，按 `Ctrl+C` 退出： diff --git a/docs/zh/prepare/index.mdx b/docs/zh/prepare/index.mdx index 7de5596..e5eef8c 100644 --- a/docs/zh/prepare/index.mdx +++ b/docs/zh/prepare/index.mdx @@ -135,6 +135,7 @@ spec: | Alauda Kafka Operator（`strimzi-kafka-operator`） | `kafka-system` | 所有命名空间 | | `opensearch-operator`（仅 OpenSearch） | `opensearch-operator` | 所有命名空间 | +- 请先安装这些 Operator，再创建步骤 3 至 5 中的资源。`log-storage-operator` 每次启动都会检查一遍命名空间：名为 `cpaas-clickhouse` 的 `ClickHouseInstallation`、名为 `cpaas-kafka` 的 `RdsKafka`、名为 `clickhouse-basic-auth` 的 `Secret`，都会被它当作更早版本 chart 安装的部署。 - 只安装缺失的 Operator。如果集群上已经装了某一个（例如由更早的版本装入），请沿用已有的，不要再装第二份：同一个 Operator 的两份实例会同时写 `cpaas-system` 下的同一批资源。此时改为检查它的 watch 范围，必要时放开。 - 不要在 `cpaas-system` 中创建 OperatorGroup。平台已在其中创建了一个，再创建一个会导致平台拒绝该命名空间下的所有 Subscription，包括平台自己的。 - `kafka-system` 和 `opensearch-operator` 的 OperatorGroup 必须不包含 `spec.targetNamespaces`。如果已经存在一个只作用于自身命名空间的 OperatorGroup，请删除该字段，并等待 Operator Pod 重启。