diff --git a/docs/en/install/index.mdx b/docs/en/install/index.mdx new file mode 100644 index 0000000..65877e0 --- /dev/null +++ b/docs/en/install/index.mdx @@ -0,0 +1,382 @@ +--- +weight: 14 +--- + +# Installation + +This chapter installs the logging components against the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service from [Environment Preparation](../prepare/index.mdx). + +Step 1 is required for ClickHouse and OpenSearch. Then run either Step 2 (ClickHouse) or Step 3 (OpenSearch): the two are mutually exclusive, and each creates the connection Secret and the `PlatformLogForward`. Step 4 is required for both storage types and deploys the two cluster plugins after `PlatformLogForward` is ready. + +## Step 1: Create the Kafka connection Secret + +**Run on the workload cluster.** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn + namespace: cpaas-system +type: Opaque +stringData: + # Required. Kafka addresses in host:port form, comma-separated for multiple brokers. + # Read from the service created in step 3 of Environment Preparation. + # Default: cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093 (SASL over TLS) + # or cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9092 (SASL without TLS) + bootstrap: "cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093" + # Required. metadata.name of the RdsKafka resource from Environment Preparation. + kafkaClusterName: "cpaas-kafka" + # Required. Name of the RdsKafkaUser from Environment Preparation. + username: "platform-logging" + # Required. The password you set in secret/platform-logging-password, key password. + # Must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts. + password: "" + # Optional. Defaults to SCRAM-SHA-512. Allowed: SCRAM-SHA-512, SCRAM-SHA-256, PLAIN. + sasl_mechanism: "SCRAM-SHA-512" + # Optional. Topic names, defaults shown. They must match the topics created in + # step 3 of Environment Preparation. + topics.log: "ALAUDA_LOG_TOPIC" + topics.event: "ALAUDA_EVENT_TOPIC" + topics.audit: "ALAUDA_AUDIT_TOPIC" + # Optional. Only for a TLS listener whose certificate the platform does not trust. + # Read the PEM from secret/-cluster-ca-cert, key ca.crt. + # Omit this whole field for the plain listener (:9092) or a system-trusted CA. + tls.ca: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +Read the values you do not have at hand: + +```bash +# Kafka password +kubectl -n cpaas-system get secret platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d + +# Kafka cluster service (bootstrap address) +kubectl -n cpaas-system get svc cpaas-kafka-kafka-bootstrap + +# Kafka CA, only when the listener uses TLS and the CA is not system-trusted +kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert \ + -o jsonpath='{.data.ca\.crt}' | base64 -d +``` + +Save the YAML as `platform-default-mq-conn.yaml` and apply it: + +```bash +kubectl apply -f platform-default-mq-conn.yaml +``` + +Do not set `tls.insecure_skip_verify: "true"`; on the plain listener omit `tls.ca` instead. + +## Step 2: ClickHouse: Install the Logging Components + +This step applies to ClickHouse only. For OpenSearch, use Step 3 instead. + +### 2.1 Create the ClickHouse connection Secret + +**Run on the workload cluster.** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn + namespace: cpaas-system +type: Opaque +stringData: + # Required. ClickHouse HTTP address including scheme and port: 8123 for HTTP, + # 8443 for HTTPS. Read it from the cluster Service created in step 4 of + # Environment Preparation: + # kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + endpoint: "http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123" + # Must match spec.configuration.clusters[0].name of the ClickHouseInstallation + # and the cluster name used by ON CLUSTER. ACP baseline default: replicated. + cluster: "replicated" + # ClickHouse database. Default: observability (from the ClickHouseInstallation). + database: "observability" + # The logging account declared in the ClickHouseInstallation. + # Default: platform-logging. + username: "platform-logging" + # Password of that account, stored in secret/clickhouse-platform-logging-password. + password: "" + # Optional. Only when the endpoint uses HTTPS with a private CA. + # Omit this whole field for the plain HTTP endpoint (:8123). + tls.ca: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +Read the values you do not have at hand: + +```bash +# ClickHouse service and port +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + +# Cluster name, database, and shard / replica counts +kubectl -n cpaas-system get chi cpaas-clickhouse \ + -o jsonpath='{.spec.configuration.clusters[0].name}{"\t"}{.spec.configuration.settings.default_database}{"\t"}{.spec.configuration.clusters[0].layout.shardsCount}{"\t"}{.spec.configuration.clusters[0].layout.replicasCount}{"\n"}' + +# Logging account password +kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d +``` + +Save the YAML as `platform-default-ch-conn.yaml` and apply it: + +```bash +kubectl apply -f platform-default-ch-conn.yaml +``` + +Do not set `tls.insecure_skip_verify: "true"`. When the endpoint uses HTTPS, provide `tls.ca` so the platform can verify the server certificate; when it uses plain HTTP, omit `tls.ca`. + +### 2.2 Create the PlatformLogForward + +**Run on the workload cluster.** + +The `output.type` field is required for ClickHouse. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + output: + type: clickhouse # Required for ClickHouse + externalStorage: + type: clickhouse # Storage type + shards: 1 # Set from the profile table below: 1 for a single node, 2 for six nodes, 3 for nine nodes + replicas: 1 # Set from the profile table below: 1 for a single node, 3 for three nodes and above + secretRef: + name: platform-default-ch-conn # Connection Secret created in Step 2.1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1 + namespace: cpaas-system +``` + +`externalStorage.shards` and `externalStorage.replicas` must match the ClickHouse profile you deployed in Environment Preparation: + +| Profile | `shards` | `replicas` | +| --- | --- | --- | +| Single node (evaluation only) | 1 | 1 | +| Three nodes | 1 | 3 | +| Six nodes | 2 | 3 | +| Nine nodes | 3 | 3 | + +A wrong value in a multi-shard or replicated deployment leaves part of the ClickHouse topology unused. + +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. These are the replica counts of the logging components themselves, not of ClickHouse: a single-node ClickHouse still runs them as they are. + +Save the YAML as `platform-log-forward.yaml`, then check whether the logging operator already created this resource on its own: + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +The operator inspects the namespace once at each start. When it finds `ClickHouseInstallation/cpaas-clickhouse`, `RdsKafka/cpaas-kafka`, or `Secret/clickhouse-basic-auth` it assumes an older chart installed them and creates `PlatformLogForward/platform-default` itself with `installMode: Adopt`. `installMode` is immutable, so applying this manifest against that object fails with `installMode is immutable`. + +When the first command prints `Adopt` and `AppRelease/logclickhouse` does not exist, nothing was adopted from a chart. Delete the object and apply this manifest in one step, then mark the two connection Secrets as yours so the operator stops deriving them: + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-ch-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +Otherwise apply the manifest: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +When `AppRelease/logclickhouse` exists, stop and use the migration chapter instead: that cluster holds chart-installed logging data. + +### 2.3 Verify + +Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column reaches `Ready` and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +`PlatformLogForward` is ready. Continue to Step 4 to deploy the cluster plugins, then run the end-to-end data check in Step 4.3. + +## Step 3: OpenSearch: Install the Logging Components + +This step applies to OpenSearch only. For ClickHouse, use Step 2 instead. + +### 3.1 Create the OpenSearch connection Secret + +**Run on the workload cluster.** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn + namespace: cpaas-system +type: Opaque +stringData: + # Required. Comma-separated HTTP(S) URLs. Put the highly available + # coordinator or load balancer first: some data paths use only the first one. + # Read it from the OpenSearch service created in step 5 of Environment + # Preparation, for example https://cpaas-opensearch.cpaas-system.svc:9200 + # or the load balancer in front of it. Do not add leading whitespace. + endpoints: "https://:9200" + # The account created in step 5 of Environment Preparation. + # Default: platform-logging. Omit both fields when the cluster allows + # anonymous access. + username: "" + password: "" +``` + +Read the values you do not have at hand: + +```bash +# OpenSearch service and port +kubectl -n cpaas-system get svc cpaas-opensearch + +# Cluster health +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' +``` + +Save the YAML as `platform-default-os-conn.yaml` and apply it: + +```bash +kubectl apply -f platform-default-os-conn.yaml +``` + +This Secret has no `tls.ca` key: the platform connects to OpenSearch with certificate verification disabled, so an endpoint behind a private CA needs no entry here. Use `https://` in `endpoints` and the connection is established regardless of the issuer. + +### 3.2 Create the PlatformLogForward + +**Run on the workload cluster.** + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + externalStorage: + type: opensearch # Storage type + secretRef: + name: platform-default-os-conn # Connection Secret created in Step 3.1 + namespace: cpaas-system + externalMessageQueue: + type: kafka # Message queue type, currently only kafka + secretRef: + name: platform-default-mq-conn # Kafka connection Secret created in Step 1 + namespace: cpaas-system +``` + +`PlatformLogForward` is cluster-scoped. Do not add `metadata.namespace` to it; the `namespace` fields inside `secretRef` still identify the connection Secrets in `cpaas-system`. The CRD defaults are `aggregateVector.replicas: 3` and `razor.replicas: 2`; set explicit values if your capacity or placement plan requires different replica counts. + +Save the YAML as `platform-log-forward.yaml`, then check whether the logging operator already created this resource on its own: + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +The operator inspects the namespace once at each start. When it finds `ClickHouseInstallation/cpaas-clickhouse`, `RdsKafka/cpaas-kafka`, or `Secret/clickhouse-basic-auth` it assumes an older chart installed them and creates `PlatformLogForward/platform-default` itself with `installMode: Adopt`. `installMode` is immutable, so applying this manifest against that object fails with `installMode is immutable`. + +When the first command prints `Adopt` and `AppRelease/logclickhouse` does not exist, nothing was adopted from a chart. Delete the object and apply this manifest in one step, then mark the connection Secrets as yours so the operator stops deriving them: + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-os-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +Otherwise apply the manifest: + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +When `AppRelease/logclickhouse` exists, stop and use the migration chapter instead: that cluster holds chart-installed logging data. + +### 3.3 Verify + +Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column reaches `Ready` and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +`PlatformLogForward` is ready. Continue to Step 4 to deploy the cluster plugins, then run the end-to-end data check in Step 4.3. + +## Step 4: Deploy the Cluster Plugins + +**Run after `PlatformLogForward/platform-default` is `Ready=True`.** + +This step is required for both ClickHouse and OpenSearch. `PlatformLogForward` installs the storage and forwarding data path; it does not install these two cluster plugins. + +### 4.1 Deploy Log Essentials + +**Run on the global cluster.** + +1. Open **Marketplace** > **Cluster Plugins** and select `global`. +2. Install **Alauda Container Platform Log Essentials** with the default configuration. +3. Verify that the plugin is running: + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=log-api + ``` + +The `STATUS` column must be `Running`. + +### 4.2 Deploy Log Collector + +Run this step on every cluster whose logs, events, and audit records must be collected. If the global cluster's own data must be collected, install it on `global` as well. + +1. Open **Marketplace** > **Cluster Plugins** and select the cluster. +2. Install **Alauda Container Platform Log Collector** and set: + +| Field | Value | +| --- | --- | +| **Log Center Plugin** | **Standard**. This is the data path managed by `log-storage-operator`; it applies to both ClickHouse and OpenSearch. Do not select the legacy `ClickHouse` or `ElasticSearch` option. | +| **Storage Cluster Name** | Enter the name of the cluster where `PlatformLogForward/platform-default` is `Ready`. When the collector and storage are in the same cluster, enter that cluster's name. This field is a text input and is not auto-discovered for the `log-storage-operator` path. | +| **Log Collector Storage Path** | Enter an absolute path for the collector's local working data. Use `/cpaas` on a traditional OS. On Alauda OS, use a writable path under `/var/cpaas`, for example `/var/cpaas`. | +| **Mount Paths** | Optional. Add absolute host paths that contain log files the collector must read. | +| **Audit**, **Event**, **Kubernetes**, **Platform**, **System**, **Workload** | Select the log types to collect. The defaults enable audit, event, system, and workload collection; Kubernetes and platform collection are disabled. | + +3. Install the plugin and verify that it is running: + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=logagent + ``` + +Run the verification command from the global cluster. The row for each collector cluster must show `STATUS=Running`. + +### 4.3 Verify the Data Path + +After both plugin checks pass, produce or locate new log, event, and audit records and confirm that they can be queried from ClickHouse or OpenSearch. If no records arrive, inspect the `Log Collector` `ModuleInfo` status and the `PlatformLogForward` conditions before changing either resource. diff --git a/docs/en/migration/index.mdx b/docs/en/migration/index.mdx new file mode 100644 index 0000000..fc4100e --- /dev/null +++ b/docs/en/migration/index.mdx @@ -0,0 +1,129 @@ +--- +weight: 16 +--- + +# Historical Data Migration + +This chapter migrates the historical log, event, and audit data that the legacy Elasticsearch storage plugin holds into the ClickHouse or OpenSearch target that the new data path uses. Run it as part of the upgrade; the upgrade chapter links to it at the point where it applies. + +## Supported scope + +| Item | Supported | +| --- | --- | +| Source | The PVCs of the legacy Elasticsearch storage plugin. Leave `source.pvcRefs` out and the operator discovers them while the legacy StatefulSet and its PVCs are still present; list them explicitly when discovery is no longer possible, for example after the plugin is uninstalled. | +| Data | The indices selected by `source.indexScope`. It accepts exact index names and globs and must match at least one index; an empty list is rejected. | +| Target | `clickhouse` or `opensearch`. `target.secretRef` uses the same connection Secret as the `PlatformLogForward`. | +| Timing | The migration boundary is captured when you create the resource, and the copy runs after the legacy plugin is uninstalled. | +| Concurrency | ClickHouse targets always run one job at a time. OpenSearch targets accept `options.maxConcurrentJobs: 2`; the default is `1`. | +| Batch size | `options.batchSize` accepts `1..100000`; `options.syncIntervalSeconds` sets the minimum wait between batches. | + +:::warning +Provide the source volumes in time. Discovery only works while the legacy StatefulSet and its PVCs are still present. If they are already gone, or if discovery fails, list the PVCs yourself in `source.pvcRefs` — the operator never guesses a source volume. Once the source PVCs are deleted, or their PVs are released and cleaned up, the historical data can no longer be migrated, so create the migration resource, or supply `pvcRefs`, before anything releases those volumes. +::: + +## Before you start + +1. The new data path is `Ready` and its `PlatformLogForward` points at the target you migrate into. The migration image for this release is available, with its complete registry, tag, or digest (`spec.image` is required). +2. **The migrated time range is still inside the retention of the target tables.** The migration does not widen, pause, or bypass the retention that the `PlatformLogForward` and razor already applied, so records older than the target TTL can be written and then removed by the normal merge. Confirm the range first, or raise the target retention and confirm the table definitions took effect, before you create this resource. + +## Step 1: Create the migration resource + +Create the `LegacyESMigration` **before** you uninstall the legacy plugin. Creating it earlier is what records the final source state and the source volumes; a resource created after the uninstall has to name the source volumes explicitly and cannot use that final capture. + +### Target OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required; migration image provided for this release, with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # Must match the PlatformLogForward target + secretRef: + name: platform-default-os-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000; default 250 + syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (no fixed wait for OpenSearch) + maxConcurrentJobs: 1 # Concurrent jobs; default 1, up to 2 for OpenSearch +``` + +### Target ClickHouse + +Change `target.type` to `clickhouse` and point the Secret at `platform-default-ch-conn`. + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # Stable migration name, used by the commands below + namespace: cpaas-system +spec: + image: # Required; migration image provided for this release, with registry and tag or digest + source: + indexScope: # Required, selects the historical indices to migrate + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # Must match the PlatformLogForward target + secretRef: + name: platform-default-ch-conn # The same connection Secret as PlatformLogForward + namespace: cpaas-system + options: + batchSize: 250 # Documents written per batch, 1~100000; default 250 + syncIntervalSeconds: 5 # Minimum wait between batches; omit to use the default (1 second for ClickHouse) + maxConcurrentJobs: 1 # Concurrent jobs; ClickHouse is always serial (1) +``` + +Save the YAML as `legacy-es-migration.yaml` and apply it: + +```bash +kubectl apply -f legacy-es-migration.yaml +``` + +## Step 2: Wait for the capture to finish + +The copy does not start until the legacy plugin is uninstalled, so the phase stops at `PrecaptureReady` for now: + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history -w +``` + +In this flow, data copy starts only after the legacy plugin is uninstalled, so the phase stays at `PrecaptureReady` until then. Continue to Step 3 only when the phase is `PrecaptureReady` (or `Succeeded` for an already completed migration). If the phase is `Blocked`, do not delete or recreate the migration resource; read the condition message and contact support. + +The new data path continues to receive log, event, and audit data while the migration runs. Keep the new data path and its target connection unchanged. If the migration does not complete, stop and contact support. + +Use the migration image provided for this ACP 4.4 Logging release. If the target data is complete but the migration stays in `Running`, stop and contact support; do not delete the migration resource or the source volumes. + +## Step 3: Verify the migration + +Once the legacy plugin is uninstalled, the worker copies the data. Watch the same resource until it succeeds: + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}' # wait for: Succeeded +``` + +If the phase is `Blocked`, do not delete or recreate the resource; read the condition message and contact support. + +Then confirm the migrated data in the target: run the same log, event, and audit queries you use in production and check that the historical range is present. + +## Retained volumes + +Keep the protected source volumes until the verification passes and the approved change window closes. `status.phase: Succeeded` is not by itself approval to release them. + diff --git a/docs/en/prepare/index.mdx b/docs/en/prepare/index.mdx new file mode 100644 index 0000000..8cf9a47 --- /dev/null +++ b/docs/en/prepare/index.mdx @@ -0,0 +1,1606 @@ +--- +weight: 13 +--- + +# Environment Preparation + +This chapter prepares the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service that the logging components use. Steps 1 to 3 apply to ClickHouse and OpenSearch alike. Step 4 applies only to ClickHouse and Step 5 only to OpenSearch: run only the one you use. + +Follow the steps in order and complete each verification. + +## Before You Start + +Make sure you have: + +1. Administrator access to the cluster that runs the logging components. +2. The nodes and disks planned in Step 1. +3. The operator packages available in the platform marketplace: `log-storage-operator`, `clickhouse-operator`, the Alauda Kafka operator, and `opensearch-operator`. + +Use [Log Component Capacity Planning](../architecture/capacity_planning.mdx) with the tables below to choose the scale, and place the workloads on dedicated nodes as described in [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). + +Run every command in this chapter from a host that has `kubectl` access to the cluster. Each YAML block is a file you save and then apply; each step gives the file name and the `kubectl apply -f` command. + +## Step 0: Choose ClickHouse or OpenSearch and the Scale + +Choose ClickHouse or OpenSearch, and the scale. Both need the Kafka service in Step 3, and Steps 4 and 5 are mutually exclusive. + +### ClickHouse profiles + +The CPU and memory values are the container limits for each ClickHouse pod. + +| Profile | ClickHouse pods | Layout | CPU limit per pod | Memory limit per pod | Measured throughput | +| --- | --- | --- | --- | --- | --- | +| Single node | 1 | 1 shard × 1 replica | 2C | 4G | 18,000 logs/s | +| Three nodes | 3 | 1 shard × 3 replicas | 2C | 4G | 20,000 logs/s | +| Six nodes | 6 | 2 shards × 3 replicas | 4C | 8G | 40,000 logs/s | +| Nine nodes | 9 | 3 shards × 3 replicas | 4C | 8G | 69,000 logs/s | + +Use the single-node profile for evaluation only, with the single-node Kafka manifest in Step 3. Start production at the three-node profile, and move to six or nine nodes when a single shard no longer fits. + +### Kafka + +Set up three brokers with a 2C/4G limit each, plus the three controllers that the manifest in Step 3 runs at 1C/2G. Size the broker volumes by retention and throughput. The single-node evaluation profile runs one broker and one controller with the same limits. + +### OpenSearch profiles + +The CPU and memory values are per-node limits. + +| Profile | Nodes | Layout | CPU limit per node | Memory limit per node | Measured throughput | +| --- | --- | --- | --- | --- | --- | +| Small scale | 3 | 3 nodes, all roles | 2C | 4G | 6,300 logs/s | +| Small scale | 5 | 5 nodes, all roles | 2C | 4G | 9,900 logs/s | +| Large scale | 3 + 5 | 3 master, 5 data | 2C master / 8C data | 4G master / 16G data | 25,000 logs/s | +| Large scale | 3 + 7 | 3 master, 7 data | 2C master / 8C data | 4G master / 16G data | 30,000 logs/s | + +The smallest supported OpenSearch profile is three nodes; do not deploy OpenSearch as a single node. Do not size below the smallest profile, and use the large-scale profiles once a single node pool can no longer serve the data volume. If your measured storage is weaker than 6,000 IOPS and 250 MB/s read/write, size up. + +### Disk + +Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, and size the volumes for your retention: 7 days for most logs, 30 days for Kubernetes logs, 180 days for events and audits, and 540 days for metering. The examples below use 200 Gi per ClickHouse pod and per Kafka broker, 20 Gi per Kafka controller, and size the OpenSearch master and data pools separately. + +## Step 1: Nodes and Disks + +1. Select the nodes that will run the storage cluster. Do not co-locate them with business workloads. +2. Label them as infra nodes and add the matching taint, following [Planning Infra Nodes for Logging Storage](../how_to/infra_nodes.mdx). The manifests below select `node-role.kubernetes.io/infra` and tolerate that taint; if your cluster uses a different key, make the same change in every manifest. +3. Attach a dedicated SSD to every node as a persistent path: + - On the traditional operating system layout, use `/cpaas/data/...`. + - On Alauda OS nodes only `/var/cpaas` is writable, so use `/var/cpaas/data/...`. +4. Make sure the path survives node re-provisioning. +5. Decide how the volumes are provisioned: + +| Approach | When to use it | What you must do | +| --- | --- | --- | +| Static local volumes | You are pinning each pod to a specific node, which is what the infra-node setup usually does | Create one StorageClass without a provisioner, and pre-create one PV per intended pod, each with `nodeAffinity` and `local.path` pointing at that pod's directory | +| Dynamic provisioner | Your platform provides a block-storage provisioner | Create the StorageClass and let the claims bind dynamically; confirm the provisioner supports `ReadWriteOnce` block volumes and the throughput above | + +For static local volumes, create one StorageClass and one PV per pod, and **reserve every PV for the claim it belongs to**. A `local` volume cannot follow its pod: unless a volume is reserved, a claim can bind a volume that was prepared for another pod or component, and after an instance is deleted and recreated its pods can bind each other's disks. Reserving the volume with `spec.claimRef` removes that risk, because the volume then only ever matches the claim named in it. + +Use a separate StorageClass per component, so one component's claims cannot bind a volume that was prepared for another. The number of PVs is the sum of the pods you plan: ClickHouse `shardsCount × replicasCount`, Kafka `replicas + controller.replicas`, and OpenSearch the sum of the node pool `replicas`. The shape of the manifest is the same for every component, and each component's step gives the exact claim name, directory, and size: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local- + labels: + # Required in a project namespace: without the grant, the pvc-validator + # admission webhook rejects every claim using this class. + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: +spec: + capacity: + storage: + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local- + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: + local: + path: + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +Repeat the `PersistentVolume` part for every pod, with a distinct `metadata.name`, a distinct `local.path` and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory. + +`capacity.storage` on a `local` volume is matching metadata, not a quota: nothing stops a pod from filling the underlying disk past it. Set it to the real usable size and enforce retention on the storage side as well. + +## Step 2: Install the Operators + +Install the operators from the platform marketplace. The `ClickHouseInstallation`, `RdsKafka`, and OpenSearch resources below and the `PlatformLogForward` used during installation all live in `cpaas-system`, so every operator must be able to reconcile resources in that namespace. Install the ClickHouse operator only for ClickHouse and the OpenSearch operator only for OpenSearch; every deployment also needs the Kafka operator and `log-storage-operator`. + +| Operator | Subscription namespace | Namespaces the operator must watch | +| --- | --- | --- | +| `log-storage-operator` | `cpaas-system` | `cpaas-system` | +| `clickhouse-operator` (ClickHouse only) | `cpaas-system` | `cpaas-system` | +| Alauda Kafka operator (`strimzi-kafka-operator`) | `kafka-system` | All namespaces | +| `opensearch-operator` (OpenSearch only) | `opensearch-operator` | All namespaces | + +- Install these operators before you create the resources in Steps 3 to 5. `log-storage-operator` inspects the namespace once at each start: it treats a `ClickHouseInstallation` named `cpaas-clickhouse`, an `RdsKafka` named `cpaas-kafka`, or a `Secret` named `clickhouse-basic-auth` as a deployment installed by an older chart. +- Install only the operators that are missing. If one is already installed on the cluster, for example by an earlier release, keep it and do not install a second copy: two copies of the same operator write to the same `cpaas-system` resources. Check its watch scope instead and widen it if needed. +- Do not create an OperatorGroup in `cpaas-system`. The platform already owns one there, and a second OperatorGroup makes the platform reject every Subscription in that namespace, including its own. +- For `kafka-system` and `opensearch-operator`, the OperatorGroup must have no `spec.targetNamespaces`. If an OperatorGroup scoped to its own namespace already exists, remove the field and wait for the operator pod to restart. + +:::warning +An operator that does not watch `cpaas-system` ignores the resources below silently: no status, no events, and no pods. Confirm the deployments are ready and that the Kafka and OpenSearch OperatorGroups reach all namespaces before you continue. +::: + +### log-storage-operator + +Required for both. It provides the `PlatformLogForward` that Installation creates. + +```bash +kubectl get crd platformlogforwards.log.alauda.io logforwards.log.alauda.io +kubectl -n cpaas-system get deploy log-storage-operator-controller-manager +kubectl -n cpaas-system get sub log-storage-operator +``` + +### ClickHouse operator + +Only for ClickHouse. The OperatorGroup already exists in `cpaas-system`, so there is nothing to patch. + +```bash +kubectl get crd clickhouseinstallations.clickhouse.altinity.com +kubectl -n cpaas-system get deploy clickhouse-operator +``` + +### Kafka operator + +Required for both. + +```bash +kubectl -n kafka-system patch operatorgroup kafka-system \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + +kubectl get crd rdskafkas.middleware.alauda.io +kubectl -n kafka-system get deploy strimzi-cluster-operator +``` + +### OpenSearch operator + +Only for OpenSearch. + +```bash +kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + +kubectl get crd opensearchclusters.opensearch.opster.io +kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager +``` + +## Step 3: Create the Kafka Service + +### 3.1 Create the SASL password Secret + +The password must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts. + +```bash +kubectl -n cpaas-system create secret generic platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 3.2 Create the broker cluster + +Apply the manifest for the profile you chose in Step 0: three nodes and above is the production profile, single node is for evaluation only. The two differ only in the replica counts and the replication factors; do not mix them. + +Do not omit these settings: + +| Setting | Why it is required | +| --- | --- | +| `message.max.bytes: "10485760"` | Audit batches are about 1.1–1.5 MiB each. The Kafka default of 1 MiB rejects every audit batch. | +| `replica.fetch.max.bytes: "10485760"` | Must be at least `message.max.bytes`, otherwise replica synchronization stalls. | +| `auto.create.topics.enable: "false"` | A mistyped topic name must not be created automatically and silently collect data. | +| `entityOperator.topicOperator` / `userOperator` | Without them, the `RdsTopic` and `RdsKafkaUser` resources in the next steps are not applied to the brokers. | + +#### 3.2.1 Three nodes and above (production) + +Create one directory per broker and per controller on the node that pod will run on, and set the ownership (uid 1001). On Alauda OS nodes replace `/cpaas` with `/var/cpaas`: + +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/broker-1 /cpaas/data/kafka/broker-2 +sudo mkdir -p /cpaas/data/kafka/controller-0 /cpaas/data/kafka/controller-1 /cpaas/data/kafka/controller-2 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + mode: KRaft + version: 4.2.0 # Minimum supported by the Alauda Kafka operator + replicas: 3 + resources: + limits: { cpu: "2", memory: 4Gi } # From the profile + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + controller: + replicas: 3 + roles: ["controller"] # Required: without it the node pool is rejected + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # Required: it creates the topics in Step 3.5 + userOperator: {} # Required: it creates the SASL user in Step 3.4 +``` + +Save the YAML as `cpaas-kafka.yaml` and apply it: + +```bash +kubectl apply -f cpaas-kafka.yaml +``` + +#### 3.2.2 Single node (evaluation only) + +Create one directory for the single broker and one for the single controller: + +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-0 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + mode: KRaft + version: 4.2.0 # Minimum supported by the Alauda Kafka operator + replicas: 1 + resources: + limits: { cpu: "2", memory: 4Gi } # From the profile + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + controller: + replicas: 1 + roles: ["controller"] # Required: without it the node pool is rejected + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # StorageClass from Step 1, for example cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # The label you set in Step 1 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "1" + min.insync.replicas: "1" + offsets.topic.replication.factor: "1" + transaction.state.log.replication.factor: "1" + transaction.state.log.min.isr: "1" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # Required: it creates the topics in Step 3.5 + userOperator: {} # Required: it creates the SASL user in Step 3.4 +``` + +Save the YAML as `cpaas-kafka.yaml` and apply it: + +```bash +kubectl apply -f cpaas-kafka.yaml +``` + +#### 3.2.3 Reserve the volumes for the broker claims (both profiles) + +The instance creates its claims immediately. They stay `Pending` until you reserve volumes for them, because the broker claim names contain a hash that is generated per instance, which is why the volumes cannot be prepared up front: + +```bash +kubectl -n cpaas-system get pvc \ + -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' +``` + +Copy every claim name — three brokers and three controllers for the three-node profile, one broker and one controller for the single-node profile — and create one pre-bound volume per claim. Set `capacity.storage` to the size the claim asks for, point `local.path` at that pod's directory, and pin the node: + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-kafka-broker-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-kafka + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-cpaas-kafka-broker--0 + local: + path: /cpaas/data/kafka/broker-0 + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +Save the YAML as `cpaas-kafka-volumes.yaml`, repeat the `PersistentVolume` part for every claim, and apply it. The claims bind as soon as their volumes exist, and the brokers and controllers start: + +```bash +kubectl apply -f cpaas-kafka-volumes.yaml +``` + +### 3.3 Wait for the broker cluster + +```bash +kubectl -n cpaas-system get rdsKafka cpaas-kafka \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' +kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka +``` + +Wait until the `Ready` condition is `True` and all broker pods are `Running`. + +Confirm the required settings reached the brokers: + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ + grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes|^log\.retention\.bytes" /tmp/strimzi.properties +``` + +`message.max.bytes` and `replica.fetch.max.bytes` must be `10485760`, and `log.retention.bytes` must match the topic cap in 3.5, otherwise stop. Read the broker values from this file: `kafka-configs.sh --describe` does not show the two static settings. + +Then check the node placement for your profile. + +#### 3.3.1 Three nodes and above (production) + +The Kafka operator applies hard pod anti-affinity, so the three brokers must land on three different nodes: + +```bash +kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +``` + +Three distinct node names are required: hard anti-affinity needs at least three schedulable nodes. + +#### 3.3.2 Single node (evaluation only) + +The single broker and the single controller share one node, so one schedulable node is enough and there is no placement check to run. + +### 3.4 Create the SASL user and its ACLs + +The logging components use a single account with access to the three topics, the consumer groups, and the broker metadata. + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser +metadata: + name: platform-logging + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + # The three topics + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } + # The consumer groups used by the log pipeline + - host: "*" + operation: All + resource: { type: group, name: alauda_log, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_event, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_audit, patternType: literal } + # The consumer group prefix used by LogForward + - host: "*" + operation: All + resource: { type: group, name: "logforward-", patternType: prefix } + # The consumer group prefix used by the log query service + - host: "*" + operation: All + resource: { type: group, name: "razor-", patternType: prefix } + # Broker metadata + - host: "*" + operation: All + resource: { type: cluster, name: kafka-cluster, patternType: literal } +``` + +Save the YAML as `platform-logging-user.yaml` and apply it: + +```bash +kubectl apply -f platform-logging-user.yaml +``` + +All nine entries are required. `operation: All` covers the read and describe permissions these entries need. Verify: + +```bash +kubectl -n cpaas-system get rdskafkauser platform-logging \ + -o jsonpath='{.status.phase}{"\n"}' # expect: Active +kubectl -n cpaas-system get secret platform-logging +``` + +The group names use underscores (`alauda_log`), while the topic names use uppercase letters and underscores (`ALAUDA_LOG_TOPIC`). + +### 3.5 Create the three topics + +The resource name must be a valid DNS name. `spec.topicName` is the broker-side name and must match the ACL entries above. + +`retention.ms` alone does not cap disk usage: one burst can fill the broker volume before the time window expires, and a full volume stops the log, event, and audit pipelines. Set `retention.bytes` as well. It is per partition, on both the topic and the broker, and is derived from the broker volume: + +``` +per-partition cap = volume size × 70% ÷ (number of topics × partitions per topic) +``` + +For the 200 Gi volume and three topics with 30 partitions each that is 200 Gi × 70% ÷ 90 = 1,670,265,059 bytes, rounded down to a whole 100 MiB segment: `1572864000`. The 30% reserve covers index files, `__consumer_offsets`, the `__cluster_metadata` replica, and the active segment that is never deleted. Use the same value for `retention.bytes` on all three topics and for `log.retention.bytes` on the brokers. Recompute it if you size the broker volume differently. + +#### 3.5.1 Three nodes and above (production) + +Create the three topics below as they are: `replicas: 3` and `min.insync.replicas: "2"`. + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC # Broker-side name; must match the ACL and the connection Secret + partitions: 30 # Upper bound for consumer parallelism + replicas: 3 + config: + retention.ms: "172800000" # 48 hours + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +``` + +Save the YAML as `alauda-topics.yaml` and apply it: + +```bash +kubectl apply -f alauda-topics.yaml +``` + +Confirm the caps were written to all three topics: + +```bash +kubectl -n cpaas-system get rdstopic \ + -o custom-columns='NAME:.metadata.name,TOPIC:.spec.topicName,PARTITIONS:.spec.partitions,RETENTION:.spec.config.retention\.bytes,SEGMENT:.spec.config.segment\.bytes' +``` + +`RETENTION` must be `1572864000` and `SEGMENT` `104857600` on every topic. `kafka-configs.sh --describe` omits the topic value when it equals the broker-level `log.retention.bytes`, so this is the reliable check. + +#### 3.5.2 Single node (evaluation only) + +Use the same three topics with `replicas: 1` and `min.insync.replicas: "1"`. + +### 3.6 Verify the Kafka service end to end + +Build the client properties inside the broker pod from the user Secret. The broker listeners require SASL, so every command below needs them. + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ + sh -c 'cat > /tmp/logging-client.properties' <----0`, and both `` and `` are zero-based. The manifest below is identical for the two profiles; only the number of PVs differs. With a dynamic provisioner, create only the StorageClass and skip the PVs. + +#### 4.1.1 Three nodes and above (production) + +The manifest in 4.4.1 is one shard and three replicas, so it needs three claims: `data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`, `-0-1-0` and `-0-2-0`. The manifest below reserves the first one: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local-clickhouse + labels: + # Required in a project namespace: without the grant, the pvc-validator + # admission webhook rejects every claim using this class. + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-clickhouse-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-clickhouse + claimRef: # Reserve this volume for exactly this claim + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + local: + path: /cpaas/data/clickhouse + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +Save the YAML as `cpaas-clickhouse-volumes.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse-volumes.yaml +``` + +Repeat the `PersistentVolume` part for the other two replicas, with a distinct `metadata.name`, a distinct `local.path` (for example `/cpaas/data/clickhouse-1` and `/cpaas/data/clickhouse-2`) and the IP of the node that pod runs on. + +#### 4.1.2 Single node (evaluation only) + +The single-node profile has one pod, so it needs one claim: `data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`. The manifest above already reserves it; apply it as it is and do not add more PVs. + +### 4.2 Create the password Secrets + +The instance defines two accounts: `admin` for administration, and `platform-logging` for the logging components. Both read their password from a Secret, so create both before you create the instance. + +```bash +kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ + --from-literal=password="$(openssl rand -hex 16)" +kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 4.3 Create the Keeper client Service + +Every profile needs a Keeper, including the single-node profile, because the logging components create `ReplicatedMergeTree` tables. How ClickHouse reaches it differs by profile. + +#### 4.3.1 Three nodes and above (production) + +The ClickHouse pods themselves form the Keeper quorum: every ClickHouse pod is also a Keeper member, and ClickHouse reaches the quorum through a headless Service that selects all ready pods of the installation. + +Save the YAML as `cpaas-clickhouse-keeper-service.yaml` and apply it: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + clusterIP: None + type: ClusterIP + ports: + - name: keeper + port: 9181 + protocol: TCP + targetPort: 9181 + selector: + clickhouse.altinity.com/chi: cpaas-clickhouse + clickhouse.altinity.com/namespace: cpaas-system + clickhouse.altinity.com/ready: "yes" + clickhouse.altinity.com/role: keeper +``` + +```bash +kubectl apply -f cpaas-clickhouse-keeper-service.yaml +``` + +The `chi`, `namespace` and `ready` labels are set by the operator. The `role: keeper` label comes from the pod template in 4.4.1. + +#### 4.3.2 Single node (evaluation only) + +The Keeper runs inside the single ClickHouse pod through the `keeper_server/*` settings in 4.4.2, and ClickHouse reaches it on `localhost`. Do not create this Service; skip to 4.4. + +### 4.4 Create the ClickHouseInstallation + +Keep the cluster name `replicated` for the logging components, and set `shardsCount` and `replicasCount` from the Step 0 profile. Apply one of the two manifests below, depending on the profile you chose. + +Replace `` with the ClickHouse server image published with the platform middleware packages, for example `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`. + +#### 4.4.1 Three nodes and above (production) + +The Keeper runs inside every ClickHouse pod, so the pods form the quorum among themselves and the installation stays a single `ClickHouseInstallation`. + +The static Keeper configuration is injected through the cluster `files` and pulls in a generated file with `include_from`. The identity-dependent part — `server_id` and the member list — is generated per pod by an init container into an in-memory `emptyDir`. Keep `SHARDS_COUNT` and `REPLICAS_COUNT` in that init container equal to `layout.shardsCount` and `layout.replicasCount`, otherwise the member list is incomplete and the quorum never forms. + +The readiness probe checks the Raft port. This is required: the default HTTP probe only succeeds after ClickHouse is serving, and ClickHouse does not finish starting until the Keeper quorum exists, so the operator would wait for the first replica forever and never create the remaining ones. + +The Keeper `path` sits under `/var/lib/clickhouse`, which is the mounted data volume, so the Keeper log and snapshots live on the persistent volume together with the ClickHouse data. Do not move it outside that mount: Keeper state kept in the container filesystem is lost whenever the pod restarts. + +The `wait-for-self-dns` init container waits until the pod resolves its own headless service name. Without it, a pod that starts before its DNS record is published initialises its distributed DDL worker against an unresolved hostname and then never retries: `CREATE TABLE ... ON CLUSTER` succeeds on the other replicas, and that replica silently misses the statement. + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # The admin password comes from the Secret created above. + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # The account the logging components use. It is declared here, like the admin + # account, so it carries the privileges the components need without a separate + # set of GRANT statements. + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper # The Service created in 4.3.1 + port: 9181 + + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + # Self-observability system tables grow without bound and eventually fill the volume. + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + + clusters: + - name: replicated # Reused in the connection Secret + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # From the profile: 1, 2, or 3 + replicasCount: 3 # 3 for three nodes and above + shards: + - files: + keeper_config.xml: | + + /tmp/clickhouse/keeper_dynamic_configuration.xml + + /var/lib/clickhouse/coordination + 9181 + * + + information + + + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + metadata: + labels: + clickhouse.altinity.com/role: keeper # Selected by the Service in 4.3.1 + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + env: + - name: RAFT_PORT + value: "9444" + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: ch-keeper + containerPort: 9181 + - name: raft + containerPort: 9444 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + - name: keeper-dynamic-config + mountPath: /tmp/clickhouse + readinessProbe: + tcpSocket: + port: 9444 + initialDelaySeconds: 10 + timeoutSeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + initContainers: + - name: wait-for-self-dns + image: + command: + - /bin/bash + - -c + - | + SELF="$(hostname -s).$(hostname -d)" + MY_IP="$(hostname -i)" + for i in $(seq 1 60); do + RESOLVED=$(getent hosts "$SELF" 2>/dev/null | awk '{print $1}' | head -1) + if [ "$RESOLVED" = "$MY_IP" ]; then + echo "resolved $SELF to $MY_IP after ${i}s" + exit 0 + fi + sleep 2 + done + echo "still unresolved after 120s, continuing" + exit 0 + - name: keeper-config-initializer + image: + env: + - name: RAFT_PORT + value: "9444" + - name: SHARDS_COUNT + value: "1" # Keep equal to layout.shardsCount + - name: REPLICAS_COUNT + value: "3" # Keep equal to layout.replicasCount + command: + - /bin/bash + - -c + - | + set -euo pipefail + OUT="/tmp/config/keeper_dynamic_configuration.xml" + HOST=$(hostname -s) + DOMAIN=$(hostname -d) + if [[ $HOST =~ (.*)-([0-9]+)-([0-9]+)-([0-9]+)$ ]]; then + SHARD=${BASH_REMATCH[2]} + REPLICA=${BASH_REMATCH[3]} + else + echo "Failed to parse shard/replica from hostname $HOST"; exit 1 + fi + if [[ $DOMAIN =~ ^(.*)-([0-9]+)-([0-9]+)\.(.*)$ ]]; then + DOMAIN_NAME=${BASH_REMATCH[1]} + DOMAIN_SUFFIX=.${BASH_REMATCH[4]} + else + echo "Failed to parse domain $DOMAIN"; exit 1 + fi + MY_ID=$((SHARD * REPLICAS_COUNT + REPLICA + 1)) + KEEPER_ID=1 + { + echo "" + echo " " + echo " ${MY_ID}" + echo " " + for (( i=0; i" + echo " ${KEEPER_ID}" + echo " ${DOMAIN_NAME}-${i}-${j}${DOMAIN_SUFFIX}" + echo " ${RAFT_PORT}" + echo " " + KEEPER_ID=$((KEEPER_ID + 1)) + done + done + echo " " + echo " " + echo "" + } > "$OUT" + echo "Keeper dynamic configuration generated for server_id=${MY_ID}" + volumeMounts: + - name: keeper-dynamic-config + mountPath: /tmp/config + volumes: + - name: keeper-dynamic-config + emptyDir: + medium: Memory + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # From 4.1 +``` + +Save the YAML as `cpaas-clickhouse.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +`default_database: observability` makes ClickHouse create the `observability` database at startup, so there is no database to create here. + +#### 4.4.2 Single node (evaluation only) + +The Keeper runs inside the ClickHouse pod through the `keeper_server/*` settings. + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # The admin password comes from the Secret created above. + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # The account the logging components use. It is declared here, like the admin + # account, so it carries the privileges the components need without a separate + # set of GRANT statements. + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + clusters: + - name: replicated # Reused in the connection Secret + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # From the profile: 1, 1, 2, or 3 + replicasCount: 1 # The example runs one pod; use 3 for three nodes and above + + settings: + default_database: observability # Reused in the connection Secret + merge_tree/materialize_ttl_recalculate_only: "1" + # Self-observability system tables grow without bound and eventually fill the volume. + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + # Co-located Keeper for the single-node profile. For three nodes and above, + # use the manifest below instead, which runs Keeper in every ClickHouse pod. + keeper_server/tcp_port: "9181" + keeper_server/server_id: "1" + keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log + keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots + keeper_server/coordination_settings/operation_timeout_ms: "10000" + keeper_server/coordination_settings/session_timeout_ms: "30000" + keeper_server/raft_configuration/server/id: "1" + keeper_server/raft_configuration/server/hostname: localhost + keeper_server/raft_configuration/server/port: "9234" + + zookeeper: + nodes: + - host: localhost + port: 9181 + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: keeper + containerPort: 9181 + - name: raft + containerPort: 9234 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # From 4.1 +``` + +Save the YAML as `cpaas-clickhouse.yaml` and apply it: + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +### 4.5 Wait for the cluster + +```bash +kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ + -o jsonpath='{.status.status}{"\n"}' # re-run until it reports: Completed + +kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse +``` + +All ClickHouse pods must be `Running` and ready, and every claim must be `Bound`. A StorageClass name that does not exist or cannot bind produces no pods and no error, so check the claims rather than the `ClickHouseInstallation` status alone. + +#### 4.5.1 Three nodes and above (production) + +Confirm the Keeper quorum before you continue. The Keeper runs inside the ClickHouse pods, so check one pod per replica: + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +``` + +The three lines must report one `leader` and two `follower`. Also confirm that ClickHouse reads the quorum through the Service from 4.3.1: + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# expect: cpaas-clickhouse-keeper 9181 +``` + +#### 4.5.2 Single node (evaluation only) + +There is one Keeper member, so it reports `standalone` rather than a leader and followers. Confirm its state and the address it listens on: + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +# expect: standalone + +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# expect: localhost 9181 +``` + +### 4.6 Read the logging account password + +The account is declared in the `ClickHouseInstallation` you applied in 4.4, so there is nothing to create here and no `GRANT` statements to run: like the `admin` account, it is a server-configuration user and carries the privileges the logging components need. + +Read its password for the connection details in Step 6. 4.7 reuses the `CH_POD` and `LOG_PASSWORD` variables, so run both in the same shell. + +```bash +CH_POD="$(kubectl -n cpaas-system get pod \ + -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o jsonpath='{.items[0].metadata.name}')" +LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "platform-logging password: $LOG_PASSWORD" +``` + +### 4.7 Verify + +Run these in the same shell, so that `$CH_POD` and `$LOG_PASSWORD` from 4.6 are still set: + +```bash +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" + +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "DROP TABLE observability.__perm_check" +``` + +Both commands must succeed; a failed create means the account cannot manage the schema. + +Record the connection values in Step 6. + +## Step 5: OpenSearch: Create the Cluster + +### 5.1 Create the cluster + +Set the node pools from the Step 0 profile. The example is 3 + 5: three master nodes and five data nodes. For the small-scale profiles, use a single pool with `roles: [cluster_manager, data]` and `replicas: 3` or `5`. + +Before you apply the manifest: + +1. Create the data directory on every node that will run an OpenSearch pod, and set its ownership (uid 1000). On Alauda OS nodes replace `/cpaas` with `/var/cpaas`: + + ```bash + sudo mkdir -p /cpaas/data/opensearch + sudo chown -R 1000:1000 /cpaas/data/opensearch + ``` + +2. Make sure `vm.max_map_count` is at least `262144`. The OpenSearch operator sets it from an init container, so nothing is required unless your cluster enforces restricted Pod Security Admission, in which case that init container cannot run and you must set it on every node yourself: + + ```bash + sudo sysctl -w vm.max_map_count=262144 + echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf + ``` + +3. With static local volumes, pre-create one pre-bound PV per node using the shape from Step 1, with `claimRef` set to `data---`. For the example below that is `data-cpaas-opensearch-masters-0` to `data-cpaas-opensearch-masters-2` and `data-cpaas-opensearch-data-0` to `data-cpaas-opensearch-data-4`. With a dynamic provisioner, skip this. + +Save the YAML as `cpaas-opensearch.yaml` and apply it: + +```yaml +apiVersion: opensearch.opster.io/v1 +kind: OpenSearchCluster +metadata: + name: cpaas-opensearch + namespace: cpaas-system +spec: + general: + serviceName: cpaas-opensearch + httpPort: 9200 + version: 3.7.0 + security: + tls: + http: + generate: true + transport: + generate: true + perNode: true + nodePools: + - component: masters + replicas: 3 + diskSize: 100Gi + roles: + - cluster_manager + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # From Step 1 + resources: + limits: + cpu: "2" # From the profile + memory: 4Gi # From the profile + requests: + cpu: "1" + memory: 2Gi + - component: data + replicas: 5 + diskSize: 800Gi + roles: + - data + - ingest + nodeSelector: + node-role.kubernetes.io/infra: "" # The label you set in Step 1 + tolerations: + - key: node-role.kubernetes.io/infra # The taint you set in Step 1 + operator: Exists + effect: NoSchedule + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # From Step 1 + resources: + limits: + cpu: "8" # From the profile + memory: 16Gi # From the profile + requests: + cpu: "2" + memory: 8Gi + dashboards: + replicas: 0 +``` + +Apply it: + +```bash +kubectl apply -f cpaas-opensearch.yaml +``` + +Wait for the cluster to be healthy: + +```bash +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' # expect: green +``` + +The health reports `unknown` and then `yellow` while the nodes start and the shards initialize. Wait for `green`. + +The steps that follow authenticate as `admin`, the security plugin administrator. The operator generates that account and stores its credentials in the `-admin-password` Secret, so read the password from there and use it wherever `` appears in the steps below. `admin` / `admin` is the default, which is what the examples assume: + +```bash +OS_ADMIN_USER="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.username}' | base64 -d)" +OS_ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "$OS_ADMIN_USER / $OS_ADMIN_PASSWORD" +``` + +Change that password before production and keep it in sync with the operator, or the operator's own health checks stop working; the platform OpenSearch knowledge base documents the procedure (write an `internal_users.yml` with the new hash to `securityconfig-secret`, create the matching `admin-credentials-secret`, then update the cluster). Replace `` with a value of at least 32 characters for the logging account. + +### 5.2 Optional: install the Chinese analyzer plugin + +`analysis-ik` is optional. Without it the standard analyzer is used and Chinese text is not segmented; with it Chinese text is segmented. + +Configure the plugin in the `OpenSearchCluster` spec. The operator passes every `pluginsList` entry to `opensearch-plugin install` each time a node starts, so the plugin survives pod restarts and node replacement. Installing it by hand inside a running container does not survive a restart, because the plugin is written outside the data volume. + +| Field | Effect | +| --- | --- | +| `spec.general.pluginsList` | Installs the plugin on every OpenSearch node | +| `spec.bootstrap.pluginsList` | Installs the plugin on the bootstrap pod that forms the cluster. Set it whenever you configure the plugin on a cluster you have not created yet, otherwise cluster initialization can fail. | + +To create the cluster with the plugin from the start, add both fields to `cpaas-opensearch.yaml` from 5.1 before you apply it: + +```yaml +spec: + general: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" + bootstrap: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" +``` + +If you already created the cluster without the plugin, patch it instead. The operator rolls the nodes to install the plugin: + +```bash +kubectl -n cpaas-system patch opensearchcluster cpaas-opensearch --type=merge -p '{"spec":{"general":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]},"bootstrap":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]}}}' +``` + +The plugin version must match `spec.general.version`. A URL that returns 404 stops every node from starting. On a cluster without external network access, host the zip on an internal server and use that URL. + +Verify that every node has the plugin: + +```bash +for p in $(kubectl -n cpaas-system get pod -l opster.io/opensearch-cluster=cpaas-opensearch -o jsonpath='{.items[*].metadata.name}'); do echo -n "$p: "; kubectl -n cpaas-system exec $p -c opensearch -- bin/opensearch-plugin list | grep -c '^analysis-ik'; done +``` + +Each node must print `1`. Then confirm the analyzer segments Chinese text. Replace `` with the cluster service address, for example `https://cpaas-opensearch.cpaas-system.svc:9200`: + +```bash +curl -sk -u "admin:" -X POST "https:///_analyze" \ + -H 'Content-Type: application/json' \ + -d '{"analyzer":"ik_smart","text":"自然语言处理技术"}' +``` + +The response must contain several tokens, for example `自然语言`, `处理`, `技术`. + +### 5.3 Create the account used by the logging components + +The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. These calls authenticate as `admin`, using the password you read in 5.1. + +```bash +OS="https://" # the cluster service, for example https://cpaas-opensearch.cpaas-system.svc:9200 + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/roles/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{ + "cluster_permissions": [ + "cluster:monitor/*", + "cluster:admin/opendistro/ism/policy/*", + "indices:admin/index_template/put", + "indices:admin/index_template/get", + "indices:admin/template/put", + "indices:admin/template/get" + ], + "index_permissions": [{ + "index_patterns": ["log-*", "event-*", "audit-*", "meter-*"], + "allowed_actions": [ + "indices:admin/create", + "indices:admin/mapping/put", + "indices:data/write/*", + "indices:data/read/*" + ] + }] + }' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/internalusers/platform-logging" \ + -H 'Content-Type: application/json' \ + -d '{"password":"","backend_roles":[]}' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/rolesmapping/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{"users":["platform-logging"]}' +``` + +The template and policy permissions are cluster-level. Declaring `indices:admin/index_template/put` and `indices:admin/template/put` under `index_permissions` does not grant them, and the logging components then fail to install their index templates. Without the role mapping the account is authenticated but has no permissions at all. + +Verify as the new account: + +```bash +curl -sk -u "platform-logging:" -X PUT "$OS/_index_template/perm-check" \ + -H 'Content-Type: application/json' \ + -d '{"index_patterns":["log-perm-check-*"],"template":{"settings":{"number_of_shards":1}}}' + +curl -sk -u "platform-logging:" -X POST "$OS/log-perm-check/_doc" \ + -H 'Content-Type: application/json' -d '{"check":1}' + +curl -sk -u "platform-logging:" "$OS/_index_template/perm-check" +``` + +All three must succeed. A `security_exception` with `no permissions for [...]` means the role or the role mapping is incomplete. Clean up with the administrator account: + +```bash +curl -sk -u "admin:" -X DELETE "$OS/_index_template/perm-check" +curl -sk -u "admin:" -X DELETE "$OS/log-perm-check" +``` + +Record the connection values in Step 6. + +The platform applies index templates with one shard and one replica. If your HA policy needs different values, apply a composable template with a higher priority and confirm the result with `GET /_index_template`. Existing indices keep the settings they were created with. + +## Step 6: Record the Connection Details + +Record the values below. The middle column is the key in the connection Secret that Installation reads the value from. + +### ClickHouse + +| Value | Connection Secret key | Where to read it | +| --- | --- | --- | +| Endpoint | `platform-default-ch-conn` → `endpoint` | The cluster Service that exposes 8123, listed by `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse`. Use `https://` and port 8443 when TLS is enabled | +| Cluster name | `cluster` | `spec.configuration.clusters[0].name` of the `ClickHouseInstallation`, which is also the name used by `ON CLUSTER`. Default: `replicated` | +| Database | `database` | `spec.configuration.settings.default_database`. Default: `observability` | +| Shards / replicas | Not in the Secret: `externalStorage.shards` / `replicas` in the `PlatformLogForward` | `spec.configuration.clusters[0].layout.shardsCount` / `replicasCount` | +| User | `username` | The logging account declared in the `ClickHouseInstallation`. Default: `platform-logging` | +| Password | `password` | `kubectl -n cpaas-system get secret clickhouse-platform-logging-password -o jsonpath='{.data.password}' \| base64 -d` | +| CA certificate | `tls.ca` | Only when the endpoint uses HTTPS with a private CA | + +### OpenSearch + +| Value | Connection Secret key | Where to read it | +| --- | --- | --- | +| Endpoint | `platform-default-os-conn` → `endpoints` | The service address or load balancer, for example `https://:9200` or `https://cpaas-opensearch.cpaas-system.svc:9200`. Put the highly available address first | +| User | `username` | The account created in 5.3. Default: `platform-logging` | +| Password | `password` | The password you set for that account | + +### Kafka + +| Value | Connection Secret key | Where to read it | +| --- | --- | --- | +| Bootstrap address | `platform-default-mq-conn` → `bootstrap` | `-kafka-bootstrap.cpaas-system.svc:9093` for SASL over TLS, or `:9092` for SASL without TLS | +| Cluster name | `kafkaClusterName` | `metadata.name` of the `RdsKafka` resource. Default: `cpaas-kafka` | +| User | `username` | The `RdsKafkaUser` name. Default: `platform-logging` | +| Password | `password` | `kubectl -n cpaas-system get secret platform-logging-password -o jsonpath='{.data.password}' \| base64 -d`. At least 32 characters on Alauda OS nodes or other FIPS-enabled hosts | +| Topics | `topics.log` / `topics.event` / `topics.audit` | `ALAUDA_LOG_TOPIC`, `ALAUDA_EVENT_TOPIC`, `ALAUDA_AUDIT_TOPIC` | +| SASL mechanism | `sasl_mechanism` | `SCRAM-SHA-512` unless the broker uses another mechanism | +| CA certificate | `tls.ca` | Only for the TLS listener on 9093: `kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' \| base64 -d` | + +## Environment Checklist + +### Common + +| Check | Expected | +| --- | --- | +| Nodes and disks | Dedicated nodes labelled and tainted, SSD mounted for persistence, and the manifests tolerate that taint | +| StorageClass | Exists for every component you install, and binds the volumes it uses | +| Operators | The ClickHouse or OpenSearch operator and the Kafka operator are ready, their CRDs exist, and they watch `cpaas-system` | +| Kafka brokers | Cluster ready, `message.max.bytes` and `replica.fetch.max.bytes` are both `10485760`, and `log.retention.bytes` matches the topic cap | +| Kafka user and ACLs | `RdsKafkaUser` is `Active`, all nine ACL entries are present | +| Kafka topics | The three topics exist on the brokers with the intended partitions, `retention.bytes`, and `segment.bytes` | +| Kafka connectivity | A record produced and consumed with the logging account | + +### ClickHouse + +| Check | Expected | +| --- | --- | +| ClickHouse | `status.status` is `Completed`, all pods ready, and the logging account can create and drop a table. Three nodes and above: the Keeper quorum reports one leader and two followers. Single node: the single Keeper member reports `standalone` | +| Keeper Service | Three nodes and above: the headless Service from 4.3.1 selects the ready pods, and `system.zookeeper_connection` points at it. Single node: no Service is created, and `system.zookeeper_connection` points at `localhost` | +| Values recorded | Endpoint, cluster name, database, shards / replicas, user, password, and the CA certificate when the endpoint uses HTTPS | + +### OpenSearch + +| Check | Expected | +| --- | --- | +| OpenSearch | Cluster health is `green`, and the logging account can manage an index template | +| Chinese analyzer | `analysis-ik` is listed on every node and segments Chinese text when you enabled it | +| Values recorded | Endpoint, user, and password | + +Fix any failed check before the logging components connect to this storage. diff --git a/docs/en/upgrade/index.mdx b/docs/en/upgrade/index.mdx new file mode 100644 index 0000000..e1b20bd --- /dev/null +++ b/docs/en/upgrade/index.mdx @@ -0,0 +1,224 @@ +--- +weight: 15 +--- + +# Upgrade + +This section explains how to upgrade **Alauda Container Platform Log Storage for Elasticsearch** in an existing ACP deployment. + +:::warning +Do not uninstall the Elasticsearch storage plugin, stop the legacy data path, or delete its PVCs, PVs, or protection finalizers before this guide tells you to, and do not stop, scale, or delete the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads. Doing it earlier can make the historical data unavailable or invalidate the migration boundary. +::: + +## Before you start + +1. The target ClickHouse or OpenSearch cluster and the new Kafka service are prepared as described in [Environment Preparation](../prepare/index.mdx), and their connection details are available. +2. The legacy Elasticsearch, Kafka, ZooKeeper, lanaya, and Razor workloads are still running, and an approved change window is in place. +3. To migrate historical data, the migration image for this release is available, with its complete registry, tag, or digest. +4. The target matches what the `PlatformLogForward` will declare. Read the deployed CRs rather than this guide: + + | Target | What must line up | + | --- | --- | + | ClickHouse | Set `externalStorage.shards` and `replicas` from the `ClickHouseInstallation` (`spec.configuration.clusters[].layout`), and set the Secret `cluster` value to the cluster name used by `ON CLUSTER`. A mismatch writes to the wrong replica set. | + | Kafka | Keep the Secret `topics.log` / `topics.event` / `topics.audit` equal to the deployed topic names and to the `KafkaUser` ACLs, and `kafkaClusterName` equal to the Kafka cluster name. The broker and topic maximum message size must accept the audit batches. | + | OpenSearch | Index shards and replicas are not controlled by the `PlatformLogForward`. The platform applies templates with 1 shard and 1 replica; apply a higher-priority template before cutover if production needs different values. | + +## Upgrade Flow at a Glance + +Use this order. Each gate must pass before the next step starts. + +| Step | Where | Action | Gate to continue | +| --- | --- | --- | --- | +| 1 | Workload cluster | Install the new data path: connection Secrets plus `PlatformLogForward` with the upgrade annotation | `Phase=Ready` and `LegacyESUpgradeCompleted` | +| 2 | Workload cluster | If history is needed, create `LegacyESMigration` as described in [Historical Data Migration](../migration/index.mdx) | `PrecaptureReady` (or `Succeeded` on an existing run) | +| 3 | Global and workload clusters | Run the controlled uninstall of the legacy plugin | `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` stay absent for 60 seconds and new log queries still pass | +| 4 | Workload cluster | Observe the same `LegacyESMigration` until it succeeds | `Phase=Succeeded` and target queries pass | +| 5 | Workload cluster | Keep the protected source volumes | Explicit approval before any cleanup | + +## Upgrade Procedure + +### Step 1: Install the new data path with the upgrade annotation + +The new data path is installed the same way as a fresh installation: create the connection Secrets for the target storage and for Kafka, then create one `PlatformLogForward`. Follow **Install Log Storage with External ClickHouse or OpenSearch** in [Installation](../install_log.mdx) for those manifests and the notes that come with them. + +**The annotation is the only difference.** Add `log.alauda.io/legacy-es-upgrade: "true"` to the `PlatformLogForward`, so the platform enters the legacy Elasticsearch upgrade flow instead of switching the log entry point directly: + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # Fixed cluster singleton name, do not change + annotations: + log.alauda.io/legacy-es-upgrade: "true" # Fixed value, enters the legacy Elasticsearch upgrade flow +spec: + installMode: Fresh # Always Fresh, do not change it to Adopt + # externalStorage and externalMessageQueue: as in Installation, pointing at the + # target storage and Kafka connection Secrets you created above +``` + +Keep `installMode: Fresh`; `Adopt` is not part of this flow. + +The new data path is not available immediately. The platform first creates it, then switches the log entry point, and finally waits for the data that queued up in the old cluster to be consumed; the time this takes depends on the backlog. Watch the status until it finishes, and press `Ctrl+C` to stop: + +```bash +kubectl get platformlogforward platform-default -w +``` + +The `Phase` column reaches `Ready`, and the `Ready` column becomes `True`. To follow the progress or troubleshoot, read the status conditions: + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +Watch the `LegacyESUpgrade` line: when the reason becomes `LegacyESUpgradeCompleted`, the log entry point has switched to the new data path and the data queued in the old cluster has been consumed, so this step is complete. If the reason is `Blocked`, the `message` explains why. + +Once this step is complete, produce or locate new log, event, and audit records, and confirm that you can query them from the new target before you continue. Do not uninstall the old plugin while `LegacyESUpgrade` is incomplete. + +If the source cluster uses the legacy Kafka, do not stop or scale Kafka, ZooKeeper, or lanaya. The platform drains the queued data through the legacy path automatically and records `LegacyKafkaDrained` when the old consumer lag reaches zero; `LegacyESUpgradeCompleted` is the gate for this procedure. + +### Step 2: Create the migration resource if history is required + +Skip this step only when the approved upgrade plan confirms that the historical Elasticsearch data is not required, and record that decision. Do not delete the source PVCs or PVs just to skip migration. + +If the data is required, create the `LegacyESMigration` resource **before** you uninstall the legacy plugin. Creating it earlier is what records the final source state and the source volumes; a resource created after the uninstall has to name the source volumes explicitly and cannot use that final capture. + +The scope the migration supports, the requirement that the migrated time range still fits the target retention, the manifests, and the verification steps are all in [Historical Data Migration](../migration/index.mdx). + +### Step 3: Uninstall the legacy Elasticsearch storage plugin + +:::warning +Run this step only when all of these conditions are true: + +- `PlatformLogForward/platform-default` is `Ready` and its `LegacyESUpgrade` condition has reason `LegacyESUpgradeCompleted`. +- If historical migration is required, `LegacyESMigration/platform-es-history` is `PrecaptureReady` or `Succeeded`. +- Confirm that the upgrade program has completed protection for the legacy Elasticsearch volumes (PVC/PV). Do not modify or remove this protection; if the platform reports a protection failure, stop and contact support. +- If the source uses legacy Kafka, the platform has confirmed that all queued data is drained. Do not stop or scale Kafka, ZooKeeper, lanaya, or Elasticsearch to force this state. +::: + +:::warning +This sequence temporarily clears platform discovery fields for Alauda Container Platform Log Storage for Elasticsearch so the legacy plugin can be removed. Run it only in the approved change window, and do not modify or delete logagent or its dependencies. +::: + +If the platform provides a supported plugin-uninstall action for this upgrade, follow the platform or support instructions first. If the action is rejected, or the platform team asks you to run the controlled procedure, use the sequence below. + +The two deletes must run back-to-back: do not wait for `ModuleInfo` to disappear, check `AppRelease`, or perform any other check between them. + +**On the global cluster: start the critical sequence** + +Set `CLUSTER` to the workload cluster name as registered in the global cluster, then run these commands: + +```bash +set -euo pipefail + +CLUSTER= + +# 1. Resolve the target ModuleInfo. Stop if the result is ambiguous. +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" +if [ "$MODULE_COUNT" -gt 1 ]; then + echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 + exit 1 +fi + +if [ "$MODULE_COUNT" -eq 1 ]; then + MODULE_INFO="$MODULE_INFOS" + MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" + MODULE_CONFIG="logcenter-${MODULE_VERSION}" + + # 2. Record the current management objects before changing them. + kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml + kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml + kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml + + # 3. Clear the discovery flags. This is also what bypasses the logagent dependency check. + kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + + # 4. Return as soon as the delete request is accepted. Do not wait here. + kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false +else + echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." +fi +``` + +**On the workload cluster: finish the critical sequence** + +Switch the current kubectl context to the workload cluster immediately. Do not run any wait or verification first. Run: + +```bash +# 5. Delete the per-cluster install record before it can recreate ModuleInfo. +kubectl delete clusterplugininstance logcenter --ignore-not-found +``` + +**Wait for the old data path to be removed** + +Switch back to the global cluster and wait for `ModuleInfo` to disappear: + +```bash +CLUSTER= +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" + +if [ -n "$MODULE_INFO" ]; then + kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No ModuleInfo remains for cluster $CLUSTER." +fi +``` + +Switch to the workload cluster and wait for the legacy `AppRelease` to be removed: + +```bash +kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +After these steps, wait 60 seconds and verify that the resources stayed gone. Each command must return no resource. + +**On the global cluster** + +```bash +kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found +``` + +**On the workload cluster** + +```bash +kubectl get clusterplugininstance logcenter --ignore-not-found +kubectl -n cpaas-system get apprelease logcenter --ignore-not-found +kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found +``` + +If `ClusterPluginInstance/logcenter` remains or reappears, delete it again and repeat the checks; until it is gone, the platform can recreate `ModuleInfo`. If `ModuleInfo` also reappears, remove `ClusterPluginInstance/logcenter` first, then delete the new `ModuleInfo` again. Do not restore the cleared discovery fields manually; verify only that `ModuleInfo`, `ClusterPluginInstance`, and `AppRelease` do not reappear. If the global-cluster block fails before `ModuleInfo` is deleted, do not run the workload-cluster block for that attempt: fix the cause and run the global sequence again. + +After the stability check passes, produce new log, event, and audit records and confirm that they can be queried from the new target. Do not continue to migration verification if the new data path is unhealthy. + +### Step 4: Retained volumes + +The protected legacy Elasticsearch PVCs and PVs are retained. Keep them until migration and target validation are complete. Do not remove protection annotations, delete PVCs or PVs, or remove finalizers. To release the volumes after validation, contact Alauda support or follow the separate approved cleanup procedure. + +Only the ES PVCs and PVs are retained by this upgrade. Legacy Kafka and ZooKeeper volumes are not part of the retention scope. + +Export or record the final PLF conditions, migration status, target query results, and retained PVC/PV names for the implementation handover. Do not remove protection annotations or delete these volumes as part of this upgrade. + +## Completion Checklist + +| Status | Expected | +| --- | --- | +| `PlatformLogForward/platform-default` | `Phase=Ready` and `LegacyESUpgradeCompleted` | +| `LegacyESMigration/platform-es-history` (if created) | `Phase=Succeeded` | +| New logging query | A new log, event, and audit record is returned by the new query path | +| Legacy plugin removal | `ModuleInfo`, `ClusterPluginInstance/logcenter`, and `AppRelease/logcenter` are absent, and the old ES workload is gone | +| Source volumes | Protected legacy ES PVCs and PVs remain present | + +If any status is not as expected, stop and contact Alauda support. Do not delete the migration resource, source volumes, or target data to work around a failure. + +## If a Step Is Blocked + +- The platform rejects the plugin removal with `moduleinfo is depended by ...`: stop, do not modify logagent, repeat the controlled sequence, and contact support if it still fails. +- Migration is `Blocked`, `Failed`, or stays `Running`: do not delete the migration resource, Jobs, source volumes, or target data; contact support. +- `ModuleInfo` or `ClusterPluginInstance` reappears: delete `ClusterPluginInstance` first, then delete the new `ModuleInfo`, and repeat the 60-second stability check. +- `AppRelease/logcenter` does not disappear: do not remove finalizers; contact support. +- New log queries fail after uninstall: stop and contact support. Do not delete the retained volumes. diff --git a/docs/zh/install/index.mdx b/docs/zh/install/index.mdx new file mode 100644 index 0000000..c9bbf7d --- /dev/null +++ b/docs/zh/install/index.mdx @@ -0,0 +1,378 @@ +--- +weight: 14 +sourceSHA: e33ff613ceb3b97d477085e52ccc4b46a9bcfa778b9d99c0e7b349057c32b24c +--- + +# 安装 + +本章把日志组件安装到[环境准备](../prepare/index.mdx)中创建的 ClickHouse 或 OpenSearch 3.7.0 集群和 Kafka 服务上。 + +无论使用 ClickHouse 还是 OpenSearch，步骤 1 都需要执行。然后执行步骤 2（ClickHouse）或步骤 3（OpenSearch）其中之一：两者互斥，各自创建连接 Secret 和 `PlatformLogForward`。步骤 4 对两种存储都需要执行，用于在 `PlatformLogForward` 就绪后部署两个集群插件。 + +## 步骤 1：创建 Kafka 连接 Secret + +**在工作负载集群执行。** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-mq-conn + namespace: cpaas-system +type: Opaque +stringData: + # 必填。Kafka 地址，格式为 host:port，多个 broker 用逗号分隔。 + # 从环境准备的步骤 3 中创建的服务读取。 + # 默认值：cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093（SASL over TLS） + # 或 cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9092（SASL 不加密） + bootstrap: "cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9093" + # 必填。环境准备中 RdsKafka 资源的 metadata.name。 + kafkaClusterName: "cpaas-kafka" + # 必填。环境准备中 RdsKafkaUser 的名称。 + username: "platform-logging" + # 必填。你在 secret/platform-logging-password 的 password 字段中设置的密码。 + # 在 Alauda OS 节点或其他启用 FIPS 的主机上必须至少 32 个字符。 + password: "" + # 选填。默认 SCRAM-SHA-512。可选值：SCRAM-SHA-512、SCRAM-SHA-256、PLAIN。 + sasl_mechanism: "SCRAM-SHA-512" + # 选填。topic 名称，默认值如下，必须与环境准备的步骤 3 中创建的 topic 一致。 + topics.log: "ALAUDA_LOG_TOPIC" + topics.event: "ALAUDA_EVENT_TOPIC" + topics.audit: "ALAUDA_AUDIT_TOPIC" + # 选填。仅当监听器启用 TLS 且平台不信任其证书时需要。 + # 从 secret/-cluster-ca-cert 的 ca.crt 字段读取 PEM 内容。 + # 使用明文监听器（:9092）或证书由系统信任时，整段删除本字段。 + tls.ca: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +需要读取的值： + +```bash +# Kafka 密码 +kubectl -n cpaas-system get secret platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d + +# Kafka 集群服务（bootstrap 地址） +kubectl -n cpaas-system get svc cpaas-kafka-kafka-bootstrap + +# Kafka CA，仅当监听器启用 TLS 且 CA 不被系统信任时需要 +kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert \ + -o jsonpath='{.data.ca\.crt}' | base64 -d +``` + +将 YAML 保存为 `platform-default-mq-conn.yaml` 并应用： + +```bash +kubectl apply -f platform-default-mq-conn.yaml +``` + +不要设置 `tls.insecure_skip_verify: "true"`；使用明文监听器时不填 `tls.ca` 即可。 + +## 步骤 2：ClickHouse：安装日志组件 + +本步骤只适用于 ClickHouse。使用 OpenSearch 时，请改用步骤 3。 + +### 2.1 创建 ClickHouse 连接 Secret + +**在工作负载集群执行。** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-ch-conn + namespace: cpaas-system +type: Opaque +stringData: + # 必填。ClickHouse HTTP 地址，包含协议和端口：HTTP 为 8123，HTTPS 为 8443。 + # 从环境准备的步骤 4 中创建的集群 Service 读取： + # kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + endpoint: "http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123" + # 必须与 ClickHouseInstallation 的 spec.configuration.clusters[0].name + # 以及 ON CLUSTER 使用的集群名一致。ACP 基线默认值：replicated。 + cluster: "replicated" + # ClickHouse 数据库。默认值：observability（来自 ClickHouseInstallation）。 + database: "observability" + # ClickHouseInstallation 中声明的日志账号。默认值：platform-logging。 + username: "platform-logging" + # 该账号的密码，存放在 secret/clickhouse-platform-logging-password 中。 + password: "" + # 选填。仅当 endpoint 使用 HTTPS 且为自签 CA 时需要。 + # 使用明文 HTTP（:8123）时，整段删除本字段。 + tls.ca: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +需要读取的值： + +```bash +# ClickHouse 服务与端口 +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse + +# 集群名、数据库、shard / replica 数量 +kubectl -n cpaas-system get chi cpaas-clickhouse \ + -o jsonpath='{.spec.configuration.clusters[0].name}{"\t"}{.spec.configuration.settings.default_database}{"\t"}{.spec.configuration.clusters[0].layout.shardsCount}{"\t"}{.spec.configuration.clusters[0].layout.replicasCount}{"\n"}' + +# 日志账号密码 +kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d +``` + +将 YAML 保存为 `platform-default-ch-conn.yaml` 并应用： + +```bash +kubectl apply -f platform-default-ch-conn.yaml +``` + +不要设置 `tls.insecure_skip_verify: "true"`。endpoint 使用 HTTPS 时必须提供 `tls.ca` 供平台校验服务端证书；使用明文 HTTP 时不填 `tls.ca`。 + +### 2.2 创建 PlatformLogForward + +**在工作负载集群执行。** + +使用 ClickHouse 时必须填写 `output.type`。 + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 集群级单例名称，固定不变 +spec: + installMode: Fresh # 固定为 Fresh，不要改成 Adopt + output: + type: clickhouse # 使用 ClickHouse 时必填 + externalStorage: + type: clickhouse # 存储类型 + shards: 1 # 按下面的规格表填写：单节点 1、六节点 2、九节点 3 + replicas: 1 # 按下面的规格表填写：单节点 1、三节点及以上 3 + secretRef: + name: platform-default-ch-conn # 步骤 2.1 创建的连接 Secret + namespace: cpaas-system + externalMessageQueue: + type: kafka # 消息队列类型，目前仅支持 kafka + secretRef: + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret + namespace: cpaas-system +``` + +`externalStorage.shards` 和 `externalStorage.replicas` 必须与环境准备中实际部署的 ClickHouse 规格一致： + +| 规格 | `shards` | `replicas` | +| --- | --- | --- | +| 单节点（仅用于验证环境） | 1 | 1 | +| 三节点 | 1 | 3 | +| 六节点 | 2 | 3 | +| 九节点 | 3 | 3 | + +多 shard 或多副本场景填错会导致 ClickHouse 的部分拓扑用不上。 + +`PlatformLogForward` 是集群级资源，不要添加 `metadata.namespace`；`secretRef` 内的 `namespace` 仍用于定位 `cpaas-system` 中的连接 Secret。CRD 默认 `aggregateVector.replicas: 3`、`razor.replicas: 2`，如果容量或部署规划要求不同副本数，请显式设置。这两个是日志组件自身的副本数，与 ClickHouse 规格无关：单节点 ClickHouse 也照常使用它们。 + +将 YAML 保存为 `platform-log-forward.yaml`，然后确认日志 Operator 是否已经自动创建了这个资源： + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +Operator 每次启动都会检查一次命名空间。只要发现 `ClickHouseInstallation/cpaas-clickhouse`、`RdsKafka/cpaas-kafka` 或 `Secret/clickhouse-basic-auth`，它就认为这是更早版本的 chart 装的，并自行创建 `PlatformLogForward/platform-default`，且 `installMode` 为 `Adopt`。`installMode` 不可修改，此时应用本文的清单会报 `installMode is immutable`。 + +如果第一条命令输出 `Adopt`，且 `AppRelease/logclickhouse` 不存在，说明并没有从 chart 接管任何数据。请先删除该对象、再应用本文清单（一条命令内完成），然后把这几个连接 Secret 标记为自己的，避免 Operator 再次派生： + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-ch-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +资源不存在时，正常应用清单： + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +如果 `AppRelease/logclickhouse` 存在，请停止操作并改用迁移章节：该集群上已有 chart 安装的日志数据。 + +### 2.3 校验 + +观察状态直到完成，按 `Ctrl+C` 退出： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 变为 `Ready`、`Ready` 列为 `True` 即为完成。查看状态条件可用于跟踪进度或排查： + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +`PlatformLogForward` 已就绪。请继续执行步骤 4 部署集群插件，然后在步骤 4.3 做端到端数据校验。 + +## 步骤 3：OpenSearch：安装日志组件 + +本步骤只适用于 OpenSearch。使用 ClickHouse 时，请改用步骤 2。 + +### 3.1 创建 OpenSearch 连接 Secret + +**在工作负载集群执行。** + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-default-os-conn + namespace: cpaas-system +type: Opaque +stringData: + # 必填。逗号分隔的 HTTP(S) URL，请把高可用 coordinator 或负载均衡地址放在第一个， + # 部分数据路径只使用第一个地址。从环境准备的步骤 5 中创建的 OpenSearch 服务读取， + # 例如 https://cpaas-opensearch.cpaas-system.svc:9200 或其前置负载均衡地址。 + # 第一个地址前不要有空格。 + endpoints: "https://:9200" + # 环境准备的步骤 5 中创建的账号，默认值：platform-logging。 + # 集群允许匿名访问时这两个字段都可以省略。 + username: "" + password: "" +``` + +需要读取的值： + +```bash +# OpenSearch 服务与端口 +kubectl -n cpaas-system get svc cpaas-opensearch + +# 集群健康状态 +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' +``` + +将 YAML 保存为 `platform-default-os-conn.yaml` 并应用： + +```bash +kubectl apply -f platform-default-os-conn.yaml +``` + +本 Secret 没有 `tls.ca` 字段：平台连接 OpenSearch 时关闭证书校验，因此使用自签 CA 的地址也无需填写。`endpoints` 使用 `https://` 即可，无论签发者是谁都能建立连接。 + +### 3.2 创建 PlatformLogForward + +**在工作负载集群执行。** + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 集群级单例名称，固定不变 +spec: + installMode: Fresh # 固定为 Fresh，不要改成 Adopt + externalStorage: + type: opensearch # 存储类型 + secretRef: + name: platform-default-os-conn # 步骤 3.1 创建的连接 Secret + namespace: cpaas-system + externalMessageQueue: + type: kafka # 消息队列类型，目前仅支持 kafka + secretRef: + name: platform-default-mq-conn # 步骤 1 创建的 Kafka 连接 Secret + namespace: cpaas-system +``` + +`PlatformLogForward` 是集群级资源，不要添加 `metadata.namespace`；`secretRef` 内的 `namespace` 仍用于定位 `cpaas-system` 中的连接 Secret。CRD 默认 `aggregateVector.replicas: 3`、`razor.replicas: 2`，如果容量或部署规划要求不同副本数，请显式设置。 + +将 YAML 保存为 `platform-log-forward.yaml`，然后确认日志 Operator 是否已经自动创建了这个资源： + +```bash +kubectl get platformlogforward platform-default -o jsonpath='{.spec.installMode}{"\n"}' +kubectl -n cpaas-system get apprelease logclickhouse +``` + +Operator 每次启动都会检查一次命名空间。只要发现 `ClickHouseInstallation/cpaas-clickhouse`、`RdsKafka/cpaas-kafka` 或 `Secret/clickhouse-basic-auth`，它就认为这是更早版本的 chart 装的，并自行创建 `PlatformLogForward/platform-default`，且 `installMode` 为 `Adopt`。`installMode` 不可修改，此时应用本文的清单会报 `installMode is immutable`。 + +如果第一条命令输出 `Adopt`，且 `AppRelease/logclickhouse` 不存在，说明并没有从 chart 接管任何数据。请先删除该对象、再应用本文清单（一条命令内完成），然后把这几个连接 Secret 标记为自己的，避免 Operator 再次派生： + +```bash +kubectl delete platformlogforward platform-default && kubectl apply -f platform-log-forward.yaml + +kubectl -n cpaas-system annotate secret platform-default-os-conn platform-default-mq-conn \ + log.alauda.io/derived-by=sre-managed --overwrite +``` + +资源不存在时，正常应用清单： + +```bash +kubectl apply -f platform-log-forward.yaml +``` + +如果 `AppRelease/logclickhouse` 存在，请停止操作并改用迁移章节：该集群上已有 chart 安装的日志数据。 + +### 3.3 校验 + +观察状态直到完成，按 `Ctrl+C` 退出： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 变为 `Ready`、`Ready` 列为 `True` 即为完成。查看状态条件可用于跟踪进度或排查： + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +`PlatformLogForward` 已就绪。请继续执行步骤 4 部署集群插件，然后在步骤 4.3 做端到端数据校验。 + +## 步骤 4：部署集群插件 + +**请在 `PlatformLogForward/platform-default` 为 `Ready=True` 后执行。** + +无论使用 ClickHouse 还是 OpenSearch，本步骤都需要执行。`PlatformLogForward` 只安装存储与转发数据链路，不会安装下面两个集群插件。 + +### 4.1 部署 Log Essentials + +**在 global 集群执行。** + +1. 打开 **Marketplace** > **Cluster Plugins**，选择 `global`。 +2. 使用默认配置安装 **Alauda Container Platform Log Essentials**。 +3. 校验插件运行状态： + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=log-api + ``` + +`STATUS` 列必须为 `Running`。 + +### 4.2 部署 Log Collector + +请在每个需要采集日志、事件和审计数据的集群上执行本步骤。如果也需要采集 global 集群自身的数据，请在 `global` 上安装。 + +1. 打开 **Marketplace** > **Cluster Plugins**，选择要安装的集群。 +2. 安装 **Alauda Container Platform Log Collector**，并填写： + +| 字段 | 取值 | +| --- | --- | +| **日志存储插件** | **Standard**。这是由 `log-storage-operator` 管理的数据链路，ClickHouse 和 OpenSearch 都选这一项。不要选择旧的 `ClickHouse` 或 `ElasticSearch` 选项。 | +| **存储集群名称** | 填写 `PlatformLogForward/platform-default` 为 `Ready` 的集群名称。采集器和存储在同一个集群时，填写该集群名称。该字段是文本输入框，`log-storage-operator` 场景不会自动发现存储集群。 | +| **日志采集器存储路径** | 填写采集器本地工作数据的绝对路径。传统操作系统使用 `/cpaas`；Alauda OS 使用 `/var/cpaas` 下可写的路径，例如 `/var/cpaas`。 | +| **挂载路径** | 选填。添加采集器需要读取日志文件的节点绝对路径。 | +| **Audit**、**Event**、**Kubernetes**、**Platform**、**System**、**Workload** | 选择需要采集的日志类型。默认启用 Audit、Event、System 和 Workload，Kubernetes 和 Platform 默认关闭。 | + +3. 安装插件并校验运行状态： + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=logagent + ``` + +请在 global 集群执行校验命令。每个采集集群对应的行都必须显示 `STATUS=Running`。 + +### 4.3 校验数据链路 + +两个插件的状态都通过后，产生或查找新的日志、事件和审计数据，确认可以从 ClickHouse 或 OpenSearch 查询到。如果没有数据到达，请先检查 `Log Collector` 的 `ModuleInfo` 状态和 `PlatformLogForward` conditions，再修改资源。 diff --git a/docs/zh/migration/index.mdx b/docs/zh/migration/index.mdx new file mode 100644 index 0000000..a189492 --- /dev/null +++ b/docs/zh/migration/index.mdx @@ -0,0 +1,129 @@ +--- +weight: 16 +sourceSHA: ec0c45131648e842846c190ed191d1ca5089a33021d8263ca0a7a3433d3a780d +--- + +# 历史数据迁移 + +本章把旧 Elasticsearch 存储插件中的历史日志、事件与审计数据迁移到新数据链路使用的 ClickHouse 或 OpenSearch 目标端。它属于升级流程的一部分，升级章节会在适用位置链接到本章。 + +## 支持范围 + +| 项 | 支持情况 | +| --- | --- | +| 源 | 旧 Elasticsearch 存储插件的 PVC。不填写 `source.pvcRefs` 时，只要旧 StatefulSet 及其 PVC 还在，Operator 会自动发现;已经无法发现时（例如旧插件已卸载）必须显式列出。 | +| 数据 | 由 `source.indexScope` 选中的索引，支持精确索引名与 glob，必须至少命中一个索引;空列表会被拒绝。 | +| 目标 | `clickhouse` 或 `opensearch`。`target.secretRef` 复用与 `PlatformLogForward` 相同的连接 Secret。 | +| 时机 | 创建资源时抓取迁移边界，拷贝在旧插件卸载之后执行。 | +| 并发 | ClickHouse 目标始终单任务执行;OpenSearch 目标可设置 `options.maxConcurrentJobs: 2`，默认为 `1`。 | +| 批次 | `options.batchSize` 取值 `1..100000`;`options.syncIntervalSeconds` 控制批次之间的最小等待。 | + +:::warning +请在源卷被释放或删除之前及时提供源卷。自动发现只在旧 StatefulSet 及其 PVC 仍然存在时有效;如果它们已经不存在,或者发现失败,请在 `source.pvcRefs` 中自行列出这些 PVC —— Operator 不会猜测源卷。一旦源端 PVC 被删除、或其 PV 被释放并清理,历史数据就无法再迁移,因此请在相关卷被释放之前创建迁移资源或提供 `pvcRefs`。 +::: + +## 开始之前 + +1. 新数据链路为 `Ready`，其 `PlatformLogForward` 指向你要迁移到的目标端;本版本的迁移镜像可用，并含完整的 registry、tag 或 digest（`spec.image` 为必填）。 +2. **迁移的时间范围必须仍在目标表的保留期内。** 迁移不会放宽、暂停或绕过 `PlatformLogForward` 与 razor 已生效的保留策略，因此早于目标 TTL 的记录可能写入后又被正常 merge 删除。请在创建该资源之前确认时间范围，或先上调目标保留期并确认表定义已生效。 + +## 步骤 1：创建迁移资源 + +请在卸载旧插件**之前**创建 `LegacyESMigration`。提前创建才会记录最终源状态与源端存储卷;卸载之后再创建，必须显式指定源卷，无法使用最终快照。 + +### 目标 OpenSearch + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # 固定的迁移名称，后续命令使用 + namespace: cpaas-system +spec: + image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest + source: + indexScope: # 必填，选择要迁移的历史索引 + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: opensearch # 必须与 PlatformLogForward 的目标一致 + secretRef: + name: platform-default-os-conn # 与 PlatformLogForward 使用同一个连接 Secret + namespace: cpaas-system + options: + batchSize: 250 # 每批写入的文档数，1~100000；默认 250 + syncIntervalSeconds: 5 # 每批之间的最小等待时间；不填则使用默认值（OpenSearch 无固定等待） + maxConcurrentJobs: 1 # 并发任务数；默认 1，OpenSearch 最多 2 +``` + +### 目标 ClickHouse + +请把 `target.type` 改为 `clickhouse`，并把 Secret 指向 `platform-default-ch-conn`。 + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: LegacyESMigration +metadata: + name: platform-es-history # 固定的迁移名称，后续命令使用 + namespace: cpaas-system +spec: + image: # 必填；本版本提供的迁移镜像，需包含 registry 和 tag 或 digest + source: + indexScope: # 必填，选择要迁移的历史索引 + - "log-workload-*" + - "log-platform-*" + - "log-system-*" + - "log-kubernetes-*" + - "event-*" + - "audit-*" + target: + type: clickhouse # 必须与 PlatformLogForward 的目标一致 + secretRef: + name: platform-default-ch-conn # 与 PlatformLogForward 使用同一个连接 Secret + namespace: cpaas-system + options: + batchSize: 250 # 每批写入的文档数，1~100000；默认 250 + syncIntervalSeconds: 5 # 每批之间的最小等待时间；不填则使用默认值（ClickHouse 为 1 秒） + maxConcurrentJobs: 1 # 并发任务数；ClickHouse 始终串行（1） +``` + +请把 YAML 保存为 `legacy-es-migration.yaml` 并应用： + +```bash +kubectl apply -f legacy-es-migration.yaml +``` + +## 步骤 2：等待抓取完成 + +拷贝要等到旧插件卸载之后才开始，因此这一步的 phase 会先停在 `PrecaptureReady`： + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history -w +``` + +在本流程中，只有旧插件卸载之后才会开始复制数据，因此在此之前 phase 会一直停留在 `PrecaptureReady`。只有 phase 为 `PrecaptureReady`（已完成的迁移为 `Succeeded`）时才继续步骤 4。如果 phase 为 `Blocked`，请勿删除或重建迁移资源；请阅读 condition message 并联系支持。 + +迁移过程中新数据链路会持续接收日志、事件和审计数据。请保持新数据链路及其目标连接不变。如果迁移无法完成，请停止并联系支持。 + +请使用本 ACP 4.4 日志版本提供的迁移镜像。如果目标数据已经完整但迁移一直停留在 `Running`，请停止并联系支持；请勿删除迁移资源或源端存储卷。 + +## 步骤 3：校验迁移 + +旧插件卸载后，worker 会开始拷贝数据。观察同一个资源直到成功： + +```bash +kubectl -n cpaas-system get legacyesmigration platform-es-history \ + -o jsonpath='{.status.phase}{"\n"}' # 期望：Succeeded +``` + +如果 phase 为 `Blocked`，请勿删除或重建该资源;请阅读 condition message 并联系支持。 + +随后在目标端确认迁移后的数据：用与生产相同的日志、事件与审计查询，确认历史时间范围的数据存在。 + +## 保留的存储卷 + +在校验通过、且变更窗口关闭之前，请保留受保护的源端存储卷。`status.phase: Succeeded` 本身不等于可以释放它们。 diff --git a/docs/zh/prepare/index.mdx b/docs/zh/prepare/index.mdx new file mode 100644 index 0000000..e5eef8c --- /dev/null +++ b/docs/zh/prepare/index.mdx @@ -0,0 +1,1606 @@ +--- +weight: 13 +sourceSHA: 2beb29dea60f229162c315ea1be37441ca745c03c69edad76669447818d40776 +--- + +# 环境准备 + +本文介绍如何准备日志组件使用的 ClickHouse 或 OpenSearch 3.7.0 集群和 Kafka 服务。步骤 1 至 3 对 ClickHouse 和 OpenSearch 都适用；步骤 4 仅用于 ClickHouse，步骤 5 仅用于 OpenSearch，请只执行你使用的那一步。 + +请按顺序执行并完成校验。 + +## 开始之前 + +请先确认： + +1. 你拥有运行日志组件的集群的管理员权限。 +2. 已按步骤 1 规划好节点和磁盘。 +3. 平台市场中已上架以下 Operator 包：`log-storage-operator`、`clickhouse-operator`、Alauda Kafka Operator、`opensearch-operator`。 + +请结合[日志组件容量规划](https://docs.alauda.cn/logging-service/4.3/architecture/capacity_planning.html)和下面的表格确定规格，并按[为日志存储规划基础设施节点](https://docs.alauda.cn/logging-service/4.3/how_to/infra_nodes.html)把工作负载放到独占节点上。 + +本章所有命令都请在能访问该集群的 `kubectl` 主机上执行。每段 YAML 都需要先保存成文件再用 `kubectl apply -f` 应用；每一步都会给出文件名和对应的 apply 命令。 + +## 步骤 0：选择 ClickHouse 或 OpenSearch 及规格 + +请先确定使用 ClickHouse 还是 OpenSearch，以及规格。两者都需要完成步骤 3 的 Kafka 服务；步骤 4 与步骤 5 互斥，只执行其中之一。 + +### ClickHouse 规格 + +CPU 和内存是每个 ClickHouse Pod 的容器 limit。 + +| 规格 | ClickHouse Pod 数 | 拓扑 | 每 Pod CPU limit | 每 Pod 内存 limit | 实测吞吐 | +| --- | --- | --- | --- | --- | --- | +| 单节点 | 1 | 1 分片 × 1 副本 | 2C | 4G | 18,000 logs/s | +| 三节点 | 3 | 1 分片 × 3 副本 | 2C | 4G | 20,000 logs/s | +| 六节点 | 6 | 2 分片 × 3 副本 | 4C | 8G | 40,000 logs/s | +| 九节点 | 9 | 3 分片 × 3 副本 | 4C | 8G | 69,000 logs/s | + +单节点规格仅用于验证环境，配合步骤 3 的单节点 Kafka 清单使用。生产环境请从三节点规格起步；单个分片无法容纳数据时，再扩展到六节点或九节点。 + +### Kafka + +请准备三个 broker，每个 limit 为 2C/4G，另外还有步骤 3 清单中三个 limit 为 1C/2G 的 controller。broker 存储量按保留时长和吞吐规划。单节点验证规格只运行一个 broker 和一个 controller，limit 与上面相同。 + +### OpenSearch 规格 + +CPU 和内存是每个节点的 limit。 + +| 规格 | 节点 | 拓扑 | 每节点 CPU limit | 每节点内存 limit | 实测吞吐 | +| --- | --- | --- | --- | --- | --- | +| 小规格 | 3 | 3 个节点，承担全部角色 | 2C | 4G | 6,300 logs/s | +| 小规格 | 5 | 5 个节点，承担全部角色 | 2C | 4G | 9,900 logs/s | +| 大规格 | 3 + 5 | 3 个 master，5 个 data | master 2C / data 8C | master 4G / data 16G | 25,000 logs/s | +| 大规格 | 3 + 7 | 3 个 master，7 个 data | master 2C / data 8C | master 4G / data 16G | 30,000 logs/s | + +OpenSearch 最小规格为三节点，不支持单节点部署。不要低于最小规格，当单个节点池无法承载数据量时，请改用大规格。如果实际存储低于 6,000 IOPS 和 250 MB/s 读写，请上调规格。 + +### 磁盘 + +请提供独占 SSD 存储，至少 6,000 IOPS 和 250 MB/s 读写，并按保留时长规划容量：大部分日志 7 天，Kubernetes 日志 30 天，事件和审计 180 天，计量 540 天。下面的示例按每个 ClickHouse Pod 200 Gi、每个 Kafka broker 200 Gi、每个 Kafka controller 20 Gi 规划；OpenSearch 的 master 与 data 节点池需要分别规划。 + +## 步骤 1：节点和磁盘 + +1. 选择用于运行存储集群的节点，不要与业务工作负载混部。 +2. 按[为日志存储规划基础设施节点](https://docs.alauda.cn/logging-service/4.3/how_to/infra_nodes.html)给这些节点打上基础设施节点标签，并添加对应的污点。下面的清单使用 `node-role.kubernetes.io/infra` 作为选择标签并容忍该污点；如果你的集群用别的 key，请在每个清单里同步改掉。 +3. 为每个节点挂载独占 SSD 作为持久化路径： + - 使用传统操作系统布局时，使用 `/cpaas/data/...`。 + - 在 Alauda OS 节点上只有 `/var/cpaas` 可写，因此使用 `/var/cpaas/data/...`。 +4. 确保该路径在节点重新纳管后仍然保留。 +5. 确定存储卷的供给方式： + +| 方式 | 适用场景 | 需要做什么 | +| --- | --- | --- | +| 静态本地卷 | 需要把每个 Pod 固定到指定节点，这也是基础设施节点方案通常采用的方式 | 创建一个不带 provisioner 的 StorageClass，并按预期的 Pod 数量预先创建 PV，每个 PV 通过 `nodeAffinity` 和 `local.path` 指向该 Pod 的目录 | +| 动态供给 | 平台已提供块存储 provisioner | 创建 StorageClass，由 PVC 动态绑定；确认 provisioner 支持 `ReadWriteOnce` 块卷并满足上述吞吐要求 | + +使用静态本地卷时，请创建一个 StorageClass 并按每个 Pod 一个 PV 预先创建，同时**把每个 PV 预留给它自己的 PVC**。`local` 卷不能跟随 Pod 迁移：如果不做预留，某个 PVC 可能绑走为其他 Pod 或其他组件准备的卷；实例删除重建后，各 Pod 之间也可能互相绑错盘。用 `spec.claimRef` 预留之后，这个卷只会匹配它上面写明的那个 PVC。 + +请为每个组件使用独立的 StorageClass，避免某个组件的 PVC 绑走为其他组件准备的卷。PV 数量等于你规划的 Pod 总数：ClickHouse 为 `shardsCount × replicasCount`，Kafka 为 `replicas + controller.replicas`，OpenSearch 为各节点池 `replicas` 之和。各组件的清单写法相同，具体的 PVC 名称、目录和容量在对应步骤中给出（Kafka 见步骤 3，ClickHouse 见步骤 4，OpenSearch 见步骤 5），把它们填进下面的模板： + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local- + labels: + # 在项目命名空间中是必需的：没有这个授权， + # pvc-validator 准入 webhook 会拒绝所有使用该 StorageClass 的 PVC。 + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: +spec: + capacity: + storage: + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local- + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: + local: + path: + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +请为每个 Pod 重复 `PersistentVolume` 部分，`metadata.name` 和 `local.path` 各不相同，`values` 填该 Pod 所在节点的 IP。同一节点上跑多个同类 Pod 时，每个 Pod 用独立目录。 + +`local` 卷上的 `capacity.storage` 只是匹配用的元数据，不是配额：Pod 完全可以写满底层磁盘。请把它设成真实可用容量，并在存储侧同时配置保留策略。 + +## 步骤 2：安装 Operator + +请从平台市场安装所需 Operator。下面创建的 `ClickHouseInstallation`、`RdsKafka` 和 OpenSearch 资源，以及安装阶段创建的 `PlatformLogForward`，都位于 `cpaas-system`，因此每个 Operator 都必须能协调该命名空间中的资源。仅 ClickHouse 需要安装 ClickHouse Operator，仅 OpenSearch 需要安装 OpenSearch Operator；两种场景都需要 Kafka Operator 和 `log-storage-operator`。 + +| Operator | Subscription 所在命名空间 | Operator 必须 watch 的命名空间 | +| --- | --- | --- | +| `log-storage-operator` | `cpaas-system` | `cpaas-system` | +| `clickhouse-operator`（仅 ClickHouse） | `cpaas-system` | `cpaas-system` | +| Alauda Kafka Operator（`strimzi-kafka-operator`） | `kafka-system` | 所有命名空间 | +| `opensearch-operator`（仅 OpenSearch） | `opensearch-operator` | 所有命名空间 | + +- 请先安装这些 Operator，再创建步骤 3 至 5 中的资源。`log-storage-operator` 每次启动都会检查一遍命名空间：名为 `cpaas-clickhouse` 的 `ClickHouseInstallation`、名为 `cpaas-kafka` 的 `RdsKafka`、名为 `clickhouse-basic-auth` 的 `Secret`，都会被它当作更早版本 chart 安装的部署。 +- 只安装缺失的 Operator。如果集群上已经装了某一个（例如由更早的版本装入），请沿用已有的，不要再装第二份：同一个 Operator 的两份实例会同时写 `cpaas-system` 下的同一批资源。此时改为检查它的 watch 范围，必要时放开。 +- 不要在 `cpaas-system` 中创建 OperatorGroup。平台已在其中创建了一个，再创建一个会导致平台拒绝该命名空间下的所有 Subscription，包括平台自己的。 +- `kafka-system` 和 `opensearch-operator` 的 OperatorGroup 必须不包含 `spec.targetNamespaces`。如果已经存在一个只作用于自身命名空间的 OperatorGroup，请删除该字段，并等待 Operator Pod 重启。 + +:::warning +Operator 如果未 watch `cpaas-system`，会静默忽略下面的资源：没有 status、没有事件、也没有 Pod。继续之前请确认 Deployment 已就绪，且 Kafka 和 OpenSearch 的 OperatorGroup 覆盖所有命名空间。 +::: + +### log-storage-operator + +两种场景都需要。安装阶段创建的 `PlatformLogForward` 由它提供。 + +```bash +kubectl get crd platformlogforwards.log.alauda.io logforwards.log.alauda.io +kubectl -n cpaas-system get deploy log-storage-operator-controller-manager +kubectl -n cpaas-system get sub log-storage-operator +``` + +### ClickHouse Operator + +仅在 ClickHouse 时需要。`cpaas-system` 中已有 OperatorGroup，无需额外操作。 + +```bash +kubectl get crd clickhouseinstallations.clickhouse.altinity.com +kubectl -n cpaas-system get deploy clickhouse-operator +``` + +### Kafka Operator + +ClickHouse 和 OpenSearch 都需要。 + +```bash +kubectl -n kafka-system patch operatorgroup kafka-system \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + +kubectl get crd rdskafkas.middleware.alauda.io +kubectl -n kafka-system get deploy strimzi-cluster-operator +``` + +### OpenSearch Operator + +仅在 OpenSearch 时需要。 + +```bash +kubectl -n opensearch-operator patch operatorgroup opensearch-operator \ + --type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]' + +kubectl get crd opensearchclusters.opensearch.opster.io +kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager +``` + +## 步骤 3：创建 Kafka 服务 + +### 3.1 创建 SASL 密码 Secret + +在 Alauda OS 节点或其他启用了 FIPS 的主机上，密码长度必须不少于 32 个字符。 + +```bash +kubectl -n cpaas-system create secret generic platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 3.2 创建 broker 集群 + +请选择与步骤 0 中选定的规格对应的一份清单：三节点及以上是生产规格，单节点仅用于验证环境。两者的区别只有副本数和副本因子，不要混用。 + +以下配置不能省略： + +| 配置项 | 为什么必须设置 | +| --- | --- | +| `message.max.bytes: "10485760"` | 审计数据的批大小约为 1.1–1.5 MiB，Kafka 默认的 1 MiB 会拒绝所有审计批次。 | +| `replica.fetch.max.bytes: "10485760"` | 必须不小于 `message.max.bytes`，否则副本同步会停滞。 | +| `auto.create.topics.enable: "false"` | 避免拼错的 topic 名称被自动创建并静默接收数据。 | +| `entityOperator.topicOperator` / `userOperator` | 缺少它们时，下一步中的 `RdsTopic` 和 `RdsKafkaUser` 不会生效到 broker。 | + +#### 3.2.1 三节点及以上（生产） + +在每台将运行 broker 或 controller 的节点上，为每个 Pod 创建一个目录并设置属主（uid 1001）。在 Alauda OS 节点上请把 `/cpaas` 换成 `/var/cpaas`： + +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/broker-1 /cpaas/data/kafka/broker-2 +sudo mkdir -p /cpaas/data/kafka/controller-0 /cpaas/data/kafka/controller-1 /cpaas/data/kafka/controller-2 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + mode: KRaft + version: 4.2.0 # Alauda Kafka Operator 支持的最低版本 + replicas: 3 + resources: + limits: { cpu: "2", memory: 4Gi } # 取自规格 + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + controller: + replicas: 3 + roles: ["controller"] # 必需：缺少时 node pool 会被拒绝 + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # 必需：由它创建 3.5 中的 topic + userOperator: {} # 必需：由它创建 3.4 中的 SASL 用户 +``` + +请把 YAML 保存为 `cpaas-kafka.yaml` 并应用： + +```bash +kubectl apply -f cpaas-kafka.yaml +``` + +#### 3.2.2 单节点（仅用于验证环境） + +只需要为唯一的 broker 和唯一的 controller 各创建一个目录： + +```bash +sudo mkdir -p /cpaas/data/kafka/broker-0 /cpaas/data/kafka/controller-0 +sudo chown -R 1001:1001 /cpaas/data/kafka +``` + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + mode: KRaft + version: 4.2.0 # Alauda Kafka Operator 支持的最低版本 + replicas: 1 + resources: + limits: { cpu: "2", memory: 4Gi } # 取自规格 + requests: { cpu: 500m, memory: 2Gi } + storage: + size: 200Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + controller: + replicas: 1 + roles: ["controller"] # 必需：缺少时 node pool 会被拒绝 + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + resources: + limits: { cpu: "1", memory: 2Gi } + requests: { cpu: 100m, memory: 512Mi } + storage: + size: 20Gi + class: # 步骤 1 的 StorageClass，例如 cpaas-local-kafka + deleteClaim: false + kafka: + template: + pod: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的标签 + operator: Exists + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + listeners: + plain: + authentication: + type: scram-sha-512 + tls: + authentication: + type: scram-sha-512 + authorization: + type: simple + config: + auto.create.topics.enable: "false" + default.replication.factor: "1" + min.insync.replicas: "1" + offsets.topic.replication.factor: "1" + transaction.state.log.replication.factor: "1" + transaction.state.log.min.isr: "1" + log.retention.hours: "48" + log.retention.bytes: "1572864000" + unclean.leader.election.enable: "false" + message.max.bytes: "10485760" + replica.fetch.max.bytes: "10485760" + socket.request.max.bytes: "104857600" + entityOperator: + topicOperator: {} # 必需：由它创建 3.5 中的 topic + userOperator: {} # 必需：由它创建 3.4 中的 SASL 用户 +``` + +请把 YAML 保存为 `cpaas-kafka.yaml` 并应用： + +```bash +kubectl apply -f cpaas-kafka.yaml +``` + +#### 3.2.3 为 broker 的 PVC 预留存储卷（两种规格通用） + +实例会立即创建自己的 PVC。在为其预留卷之前，这些 PVC 会一直处于 `Pending`：broker 的 PVC 名称里含有按实例生成的 hash，所以卷无法提前准备： + +```bash +kubectl -n cpaas-system get pvc \ + -o custom-columns='PVC:.metadata.name,STATUS:.status.phase,CLASS:.spec.storageClassName' +``` + +把每个 PVC 名称原样抄下来（三节点规格为三个 broker、三个 controller；单节点规格为一个 broker、一个 controller），并为每个 PVC 创建一个预绑定的卷。`capacity.storage` 填该 PVC 申请的容量，`local.path` 指向该 Pod 的目录，并用节点亲和把它钉在对应节点上： + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-kafka-broker-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-kafka + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-cpaas-kafka-broker--0 + local: + path: /cpaas/data/kafka/broker-0 + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +请把 YAML 保存为 `cpaas-kafka-volumes.yaml`，为全部 PVC 重复 `PersistentVolume` 部分后应用。卷一旦存在，PVC 立即绑定，broker 与 controller 随之启动： + +```bash +kubectl apply -f cpaas-kafka-volumes.yaml +``` + +### 3.3 等待 broker 集群就绪 + +```bash +kubectl -n cpaas-system get rdsKafka cpaas-kafka \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}' +kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka +``` + +等待 `Ready` 条件变为 `True`，且所有 broker Pod 处于 `Running`。 + +确认必需配置已下发到 broker： + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \ + grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes|^log\.retention\.bytes" /tmp/strimzi.properties +``` + +`message.max.bytes` 和 `replica.fetch.max.bytes` 必须是 `10485760`，`log.retention.bytes` 必须与 3.5 中的 topic 上限一致，否则不要继续。broker 侧取值以本文件为准：`kafka-configs.sh --describe` 不会显示这两个静态配置。 + +然后按规格检查节点分布。 + +#### 3.3.1 三节点及以上（生产） + +Kafka Operator 会自动施加硬反亲和，因此三个 broker 必须落在三个不同节点上： + +```bash +kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' +``` + +必须出现三个不同的节点名：硬反亲和要求集群至少有 3 个可调度节点。 + +#### 3.3.2 单节点（仅用于验证环境） + +唯一的 broker 和唯一的 controller 共用一个节点，一个可调度节点即可，不需要额外检查。 + +### 3.4 创建 SASL 用户及其 ACL + +日志组件使用同一个账号，需要访问三个 topic、消费组以及 broker 元数据。 + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser +metadata: + name: platform-logging + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + # 三个 topic + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal } + - host: "*" + operation: All + resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal } + # 日志链路使用的消费组 + - host: "*" + operation: All + resource: { type: group, name: alauda_log, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_event, patternType: literal } + - host: "*" + operation: All + resource: { type: group, name: alauda_audit, patternType: literal } + # LogForward 使用的消费组前缀 + - host: "*" + operation: All + resource: { type: group, name: "logforward-", patternType: prefix } + # 日志查询服务使用的消费组前缀 + - host: "*" + operation: All + resource: { type: group, name: "razor-", patternType: prefix } + # broker 元数据 + - host: "*" + operation: All + resource: { type: cluster, name: kafka-cluster, patternType: literal } +``` + +请把 YAML 保存为 `platform-logging-user.yaml` 并应用： + +```bash +kubectl apply -f platform-logging-user.yaml +``` + +这九条都必须配置，`operation: All` 已覆盖这些条目所需的读和 describe 权限。请校验： + +```bash +kubectl -n cpaas-system get rdskafkauser platform-logging \ + -o jsonpath='{.status.phase}{"\n"}' # 期望：Active +kubectl -n cpaas-system get secret platform-logging +``` + +消费组名称使用下划线（`alauda_log`），而 topic 名称使用大写字母和下划线（`ALAUDA_LOG_TOPIC`）。 + +### 3.5 创建三个 topic + +资源名必须是合法的 DNS 名称。`spec.topicName` 是 broker 侧名称，必须与上面的 ACL 条目一致。 + +只配置 `retention.ms` 不能限制磁盘占用：一波流量峰值可能在时间窗口到期前就把 broker 卷写满，而卷写满会导致日志、事件、审计三条链路全部停摆，因此必须同时配置 `retention.bytes`。它对 topic 和 broker 都是按分区生效的，取值由 broker 卷容量推导： + +``` +单分区上限 = 卷容量 × 70% ÷（topic 数 × 每 topic 分区数） +``` + +以 200 Gi 卷、3 个 topic、每个 topic 30 分区为例：200 Gi × 70% ÷ 90 = 1,670,265,059 字节，向下取整到完整的 100 MiB segment 后为 `1572864000`。预留的 30% 用于索引文件、`__consumer_offsets`、`__cluster_metadata` 副本，以及永远不会被删除的活跃 segment。三个 topic 的 `retention.bytes` 和 broker 的 `log.retention.bytes` 都要使用同一个值；如果 broker 卷容量不同，请重新计算。 + +#### 3.5.1 三节点及以上（生产） + +下面三个 topic 直接使用 `replicas: 3` 和 `min.insync.replicas: "2"`： + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC # broker 侧名称，必须与 ACL 和连接 Secret 一致 + partitions: 30 # 消费并发度的上限 + replicas: 3 + config: + retention.ms: "172800000" # 48 小时 + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + segment.bytes: "104857600" + retention.bytes: "1572864000" + min.insync.replicas: "2" + compression.type: producer + max.message.bytes: "10485760" +``` + +请把 YAML 保存为 `alauda-topics.yaml` 并应用： + +```bash +kubectl apply -f alauda-topics.yaml +``` + +确认三个 topic 都已写入容量上限： + +```bash +kubectl -n cpaas-system get rdstopic \ + -o custom-columns='NAME:.metadata.name,TOPIC:.spec.topicName,PARTITIONS:.spec.partitions,RETENTION:.spec.config.retention\.bytes,SEGMENT:.spec.config.segment\.bytes' +``` + +每个 topic 的 `RETENTION` 必须是 `1572864000`，`SEGMENT` 必须是 `104857600`。当 topic 的值与 broker 级 `log.retention.bytes` 相同时，`kafka-configs.sh --describe` 不会显示该值，因此以这里的检查为准。 + +#### 3.5.2 单节点（仅用于验证环境） + +沿用上面三个 topic，把 `replicas` 改为 `1`，`min.insync.replicas` 改为 `"1"`。 + +### 3.6 端到端校验 Kafka 服务 + +先在 broker Pod 内根据用户 Secret 生成客户端配置。broker 监听端口要求 SASL 认证，下面的命令都需要它。 + +```bash +KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \ + -l strimzi.io/cluster=cpaas-kafka,strimzi.io/broker-role=true \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \ + sh -c 'cat > /tmp/logging-client.properties' <----0`，`` 和 `` 都从 0 开始编号。下面的清单两种规格完全相同，区别只在 PV 数量。使用动态供给时只需创建 StorageClass，跳过 PV。 + +#### 4.1.1 三节点及以上（生产） + +4.4.1 的清单是 1 shard × 3 replica，需要三个 PVC：`data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`、`-0-1-0` 和 `-0-2-0`。下面的清单先预留第一个： + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: cpaas-local-clickhouse + labels: + # 在项目命名空间中是必需的：没有这个授权， + # pvc-validator 准入 webhook 会拒绝所有使用该 StorageClass 的 PVC。 + project.cpaas.io/ALL_ALL: "true" +provisioner: kubernetes.io/no-provisioner +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Retain +allowVolumeExpansion: false +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: cpaas-clickhouse-0 +spec: + capacity: + storage: 200Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: cpaas-local-clickhouse + claimRef: # 该卷只留给下面这个 PVC + apiVersion: v1 + kind: PersistentVolumeClaim + namespace: cpaas-system + name: data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0 + local: + path: /cpaas/data/clickhouse + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [""] +``` + +请把 YAML 保存为 `cpaas-clickhouse-volumes.yaml` 并应用： + +```bash +kubectl apply -f cpaas-clickhouse-volumes.yaml +``` + +请为另外两个副本重复 `PersistentVolume` 部分，`metadata.name` 各不相同，`local.path` 用各自的目录（例如 `/cpaas/data/clickhouse-1`、`/cpaas/data/clickhouse-2`），`values` 填该 Pod 所在节点的 IP。 + +#### 4.1.2 单节点（仅用于验证环境） + +单节点规格只有一个 Pod，只需要一个 PVC：`data-volumeclaim-template-chi-cpaas-clickhouse-replicated-0-0-0`。上面的清单已经预留了它，直接应用即可，不要再创建额外的 PV。 + +### 4.2 创建密码 Secret + +实例定义了两个账号：用于管理的 `admin`，以及供日志组件使用的 `platform-logging`。两者的密码都来自 Secret，因此请在创建实例之前先把两个 Secret 建好。 + +```bash +kubectl -n cpaas-system create secret generic clickhouse-basic-auth \ + --from-literal=password="$(openssl rand -hex 16)" +kubectl -n cpaas-system create secret generic clickhouse-platform-logging-password \ + --from-literal=password="$(openssl rand -hex 16)" +``` + +### 4.3 创建 Keeper 客户端 Service + +所有规格都需要 Keeper，单节点也一样，因为日志组件创建的是 `ReplicatedMergeTree` 表；两种规格的区别在于 ClickHouse 如何访问它。 + +#### 4.3.1 三节点及以上（生产） + +三节点及以上时，由 ClickHouse Pod 自身组成 Keeper 仲裁集群：每个 ClickHouse Pod 同时也是一个 Keeper 成员。ClickHouse 通过一个 headless Service 访问该仲裁集群，该 Service 会选中本次安装中所有就绪的 Pod。 + +请把 YAML 保存为 `cpaas-clickhouse-keeper-service.yaml` 并应用： + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: cpaas-clickhouse-keeper + namespace: cpaas-system +spec: + clusterIP: None + type: ClusterIP + ports: + - name: keeper + port: 9181 + protocol: TCP + targetPort: 9181 + selector: + clickhouse.altinity.com/chi: cpaas-clickhouse + clickhouse.altinity.com/namespace: cpaas-system + clickhouse.altinity.com/ready: "yes" + clickhouse.altinity.com/role: keeper +``` + +```bash +kubectl apply -f cpaas-clickhouse-keeper-service.yaml +``` + +其中 `chi`、`namespace`、`ready` 三个标签由 Operator 打上，`role: keeper` 标签来自 4.4.1 的 pod template。 + +#### 4.3.2 单节点（仅用于验证环境） + +单节点规格下，Keeper 通过 4.4.2 中的 `keeper_server/*` 配置运行在唯一的 ClickHouse Pod 内，ClickHouse 通过 `localhost` 访问它。不要创建这个 Service，直接进入 4.4。 + +### 4.4 创建 ClickHouseInstallation + +cluster 名称固定为 `replicated`，需与日志组件保持一致；`shardsCount` 和 `replicasCount` 按步骤 0 的规格设置。下面两份清单请按你选的规格二选一执行。 + +请把 `` 替换为平台 middleware 包发布的 ClickHouse server 镜像，例如 `registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e`。 + +#### 4.4.1 三节点及以上（生产） + +生产环境请使用这份清单。Keeper 运行在每一个 ClickHouse Pod 内，Pod 之间自行组成仲裁集群，因此整个实例仍然只是一个 `ClickHouseInstallation`。 + +静态 Keeper 配置通过 cluster 的 `files` 注入，并用 `include_from` 引入生成出来的文件；与身份相关的部分（`server_id` 和成员列表）由 init 容器按 Pod 生成到内存 `emptyDir` 中。init 容器里的 `SHARDS_COUNT`、`REPLICAS_COUNT` 必须与 `layout.shardsCount`、`layout.replicasCount` 保持一致，否则成员列表不完整，仲裁永远组不起来。 + +readiness 探针探测的是 Raft 端口，这是必需的：默认的 HTTP 探针要等 ClickHouse 开始提供服务才会成功，而 ClickHouse 又必须等 Keeper 仲裁集群就绪才能完成启动，于是 Operator 会一直等第一个副本，永远不创建其余副本。 + +Keeper 的 `path` 位于 `/var/lib/clickhouse` 之下，也就是挂载的数据卷内，因此 Keeper 的日志与快照和 ClickHouse 数据一样保存在持久卷上。不要把它移到该挂载点之外：放在容器文件系统里的 Keeper 状态会在 Pod 每次重启时丢失。 + +`wait-for-self-dns` init 容器会等待 Pod 能解析自己的 headless Service 名称。没有它时，如果某个 Pod 在自己的 DNS 记录发布之前启动，它的分布式 DDL worker 会基于一个解析不了的主机名完成初始化并且不再重试：`CREATE TABLE ... ON CLUSTER` 在其他副本上执行成功，而该副本会静默漏掉这条语句。 + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # 管理员密码来自上面创建的 Secret + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # 日志组件使用的账号。与 admin 一样在这里声明， + # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + zookeeper: + nodes: + - host: cpaas-clickhouse-keeper # 4.3.1 中创建的 Service + port: 9181 + + settings: + default_database: observability # 连接 Secret 中需要复用该名称 + merge_tree/materialize_ttl_recalculate_only: "1" + # 自身可观测性系统表会无限增长，最终写满数据盘。 + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + + clusters: + - name: replicated # 连接 Secret 中需要复用该名称 + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # 取自规格：1、2 或 3 + replicasCount: 3 # 三节点及以上为 3 + shards: + - files: + keeper_config.xml: | + + /tmp/clickhouse/keeper_dynamic_configuration.xml + + /var/lib/clickhouse/coordination + 9181 + * + + information + + + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + metadata: + labels: + clickhouse.altinity.com/role: keeper # 由 4.3.1 的 Service 选中 + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + env: + - name: RAFT_PORT + value: "9444" + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: ch-keeper + containerPort: 9181 + - name: raft + containerPort: 9444 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + - name: keeper-dynamic-config + mountPath: /tmp/clickhouse + readinessProbe: + tcpSocket: + port: 9444 + initialDelaySeconds: 10 + timeoutSeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + initContainers: + - name: wait-for-self-dns + image: + command: + - /bin/bash + - -c + - | + SELF="$(hostname -s).$(hostname -d)" + MY_IP="$(hostname -i)" + for i in $(seq 1 60); do + RESOLVED=$(getent hosts "$SELF" 2>/dev/null | awk '{print $1}' | head -1) + if [ "$RESOLVED" = "$MY_IP" ]; then + echo "resolved $SELF to $MY_IP after ${i}s" + exit 0 + fi + sleep 2 + done + echo "still unresolved after 120s, continuing" + exit 0 + - name: keeper-config-initializer + image: + env: + - name: RAFT_PORT + value: "9444" + - name: SHARDS_COUNT + value: "1" # 与 layout.shardsCount 保持一致 + - name: REPLICAS_COUNT + value: "3" # 与 layout.replicasCount 保持一致 + command: + - /bin/bash + - -c + - | + set -euo pipefail + OUT="/tmp/config/keeper_dynamic_configuration.xml" + HOST=$(hostname -s) + DOMAIN=$(hostname -d) + if [[ $HOST =~ (.*)-([0-9]+)-([0-9]+)-([0-9]+)$ ]]; then + SHARD=${BASH_REMATCH[2]} + REPLICA=${BASH_REMATCH[3]} + else + echo "Failed to parse shard/replica from hostname $HOST"; exit 1 + fi + if [[ $DOMAIN =~ ^(.*)-([0-9]+)-([0-9]+)\.(.*)$ ]]; then + DOMAIN_NAME=${BASH_REMATCH[1]} + DOMAIN_SUFFIX=.${BASH_REMATCH[4]} + else + echo "Failed to parse domain $DOMAIN"; exit 1 + fi + MY_ID=$((SHARD * REPLICAS_COUNT + REPLICA + 1)) + KEEPER_ID=1 + { + echo "" + echo " " + echo " ${MY_ID}" + echo " " + for (( i=0; i" + echo " ${KEEPER_ID}" + echo " ${DOMAIN_NAME}-${i}-${j}${DOMAIN_SUFFIX}" + echo " ${RAFT_PORT}" + echo " " + KEEPER_ID=$((KEEPER_ID + 1)) + done + done + echo " " + echo " " + echo "" + } > "$OUT" + echo "Keeper dynamic configuration generated for server_id=${MY_ID}" + volumeMounts: + - name: keeper-dynamic-config + mountPath: /tmp/config + volumes: + - name: keeper-dynamic-config + emptyDir: + medium: Memory + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # 来自 4.1 +``` + +请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +`default_database: observability` 会让 ClickHouse 在启动时创建 `observability` 库，这里不需要再建库。 + +#### 4.4.2 单节点（仅用于验证环境） + +Keeper 通过 `keeper_server/*` 配置运行在 ClickHouse Pod 内。 + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + configuration: + users: + # 管理员密码来自上面创建的 Secret + admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password + admin/networks/ip: + - "0.0.0.0/0" + - "::/0" + admin/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + # 日志组件使用的账号。与 admin 一样在这里声明， + # 因此它自带组件所需的全部权限，不需要额外的 GRANT 语句。 + platform-logging/k8s_secret_password: cpaas-system/clickhouse-platform-logging-password/password + platform-logging/networks/ip: + - "0.0.0.0/0" + - "::/0" + platform-logging/profile: default + platform-logging/quota: default + platform-logging/grants/query: + - GRANT ALL ON *.* WITH GRANT OPTION + + profiles: + default/allow_nondeterministic_mutations: "1" + default/allow_unrestricted_reads_from_keeper: "1" + default/max_execution_time: 120 + default/max_estimated_execution_time: 120 + + clusters: + - name: replicated # 连接 Secret 中需要复用该名称 + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + layout: + shardsCount: 1 # 取自规格：1、1、2 或 3 + replicasCount: 1 # 示例为单 Pod；三节点及以上请改为 3 + + settings: + default_database: observability # 连接 Secret 中需要复用该名称 + merge_tree/materialize_ttl_recalculate_only: "1" + # 自身可观测性系统表会无限增长，最终写满数据盘。 + asynchronous_metric_log/database: system + asynchronous_metric_log/table: asynchronous_metric_log + asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + metric_log/database: system + metric_log/table: metric_log + metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + trace_log/database: system + trace_log/table: trace_log + trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE" + # 单节点规格使用同 Pod 内嵌 Keeper。三节点及以上请改用下面那份清单， + # 它会为每个 ClickHouse Pod 都运行一个 Keeper。 + keeper_server/tcp_port: "9181" + keeper_server/server_id: "1" + keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log + keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots + keeper_server/coordination_settings/operation_timeout_ms: "10000" + keeper_server/coordination_settings/session_timeout_ms: "30000" + keeper_server/raft_configuration/server/id: "1" + keeper_server/raft_configuration/server/hostname: localhost + keeper_server/raft_configuration/server/port: "9234" + + zookeeper: + nodes: + - host: localhost + port: 9181 + + defaults: + templates: + podTemplate: pod-template + dataVolumeClaimTemplate: data-volumeclaim-template + serviceTemplate: service-template + + templates: + podTemplates: + - name: pod-template + podDistribution: + - scope: Shard + topologyKey: kubernetes.io/hostname + type: ShardAntiAffinity + spec: + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + containers: + - name: clickhouse + image: + ports: + - name: http + containerPort: 8123 + - name: client + containerPort: 9000 + - name: interserver + containerPort: 9009 + - name: keeper + containerPort: 9181 + - name: raft + containerPort: 9234 + resources: + requests: + cpu: "1" + memory: 4Gi + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + volumeMounts: + - name: data-volumeclaim-template + mountPath: /var/lib/clickhouse + + serviceTemplates: + - name: service-template + spec: + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + type: ClusterIP + + volumeClaimTemplates: + - name: data-volumeclaim-template + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 200Gi + storageClassName: # 来自 4.1 +``` + +请把 YAML 保存为 `cpaas-clickhouse.yaml` 并应用： + +```bash +kubectl apply -f cpaas-clickhouse.yaml +``` + +### 4.5 等待集群就绪 + +```bash +kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \ + -o jsonpath='{.status.status}{"\n"}' # 反复执行，直到变为：Completed + +kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse +kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse +``` + +所有 ClickHouse Pod 必须处于 `Running` 且就绪，且每个 PVC 都必须是 `Bound`。StorageClass 不存在或无法绑定时不会有任何 Pod，也不会有报错，因此不能只看 `ClickHouseInstallation` 的 status。 + +#### 4.5.1 三节点及以上（生产） + +请先确认 Keeper 仲裁集群再继续。Keeper 运行在 ClickHouse Pod 内，因此逐个副本检查一个 Pod： + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +``` + +三条命令的结果必须是 1 个 `leader`、2 个 `follower`。同时确认 ClickHouse 是通过 4.3.1 的 Service 访问仲裁集群的： + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# 期望：cpaas-clickhouse-keeper 9181 +``` + +#### 4.5.2 单节点（仅用于验证环境） + +单节点只有一个 Keeper 成员，状态是 `standalone`，不存在 leader 与 follower。确认它的状态和监听地址： + +```bash +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state +# 期望：standalone + +kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \ + clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV" +# 期望：localhost 9181 +``` + +### 4.6 读取日志账号密码 + +该账号已在 4.4 应用的 `ClickHouseInstallation` 中声明，因此这里不需要创建账号，也没有 `GRANT` 语句要执行：它和 `admin` 一样是服务端配置用户，自带日志组件所需的权限。 + +请读取它的密码，供步骤 6 记录连接信息。4.7 会复用 `CH_POD` 与 `LOG_PASSWORD` 变量，因此两步请在同一个 shell 中执行。 + +```bash +CH_POD="$(kubectl -n cpaas-system get pod \ + -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o jsonpath='{.items[0].metadata.name}')" +LOG_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-platform-logging-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "platform-logging password: $LOG_PASSWORD" +``` + +### 4.7 校验 + +请在同一个 shell 中执行（4.6 定义的 `$CH_POD` 和 `$LOG_PASSWORD` 还在）： + +```bash +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory" + +kubectl -n cpaas-system exec "$CH_POD" -- \ + clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \ + --query "DROP TABLE observability.__perm_check" +``` + +两条命令都必须成功，建表失败说明该账号无法管理表结构。 + +连接信息请在步骤 6 记录。 + +## 步骤 5：OpenSearch：创建集群 + +### 5.1 创建集群 + +按步骤 0 的规格设置节点池。示例为 3 + 5：三个 master、五个 data。小规格只用一个节点池，`roles` 设为 `[cluster_manager, data]`，`replicas` 设为 3 或 5。 + +应用清单前请先完成： + +1. 在每台将运行 OpenSearch Pod 的节点上创建数据目录并设置属主（uid 1000）。在 Alauda OS 节点上请把 `/cpaas` 换成 `/var/cpaas`： + + ```bash + sudo mkdir -p /cpaas/data/opensearch + sudo chown -R 1000:1000 /cpaas/data/opensearch + ``` + +2. 确认 `vm.max_map_count` 不小于 `262144`。OpenSearch Operator 会通过 init 容器设置它，因此通常不用处理；只有集群启用了受限的 Pod Security Admission 时，该 init 容器无法设置，才需要在每个节点上手工设置： + + ```bash + sudo sysctl -w vm.max_map_count=262144 + echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf + ``` + +3. 使用静态本地卷时，需要按步骤 1 的模板为每个节点预先创建预绑定 PV，`claimRef` 填 `data---`。对应下面示例就是 `data-cpaas-opensearch-masters-0` 到 `data-cpaas-opensearch-masters-2`，以及 `data-cpaas-opensearch-data-0` 到 `data-cpaas-opensearch-data-4`。使用动态供给时跳过本项。 + +请把 YAML 保存为 `cpaas-opensearch.yaml` 并应用： + +```yaml +apiVersion: opensearch.opster.io/v1 +kind: OpenSearchCluster +metadata: + name: cpaas-opensearch + namespace: cpaas-system +spec: + general: + serviceName: cpaas-opensearch + httpPort: 9200 + version: 3.7.0 + security: + tls: + http: + generate: true + transport: + generate: true + perNode: true + nodePools: + - component: masters + replicas: 3 + diskSize: 100Gi + roles: + - cluster_manager + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # 来自步骤 1 + resources: + limits: + cpu: "2" # 取自规格 + memory: 4Gi # 取自规格 + requests: + cpu: "1" + memory: 2Gi + - component: data + replicas: 5 + diskSize: 800Gi + roles: + - data + - ingest + nodeSelector: + node-role.kubernetes.io/infra: "" # 步骤 1 中设置的标签 + tolerations: + - key: node-role.kubernetes.io/infra # 步骤 1 中设置的污点 + operator: Exists + effect: NoSchedule + persistence: + pvc: + accessModes: + - ReadWriteOnce + storageClass: # 来自步骤 1 + resources: + limits: + cpu: "8" # 取自规格 + memory: 16Gi # 取自规格 + requests: + cpu: "2" + memory: 8Gi + dashboards: + replicas: 0 +``` + +应用： + +```bash +kubectl apply -f cpaas-opensearch.yaml +``` + +等待集群健康： + +```bash +kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \ + -o jsonpath='{.status.health}{"\n"}' # 期望：green +``` + +节点启动和分片初始化的过程中，健康状态会先显示 `unknown`，随后是 `yellow`。请等到 `green`。 + +后文的这些调用都使用管理员账号 `admin` 认证。该账号由 Operator 创建，凭据保存在 `-admin-password` Secret 中，请从这里读取密码，填到后文所有 `` 的位置。默认账号密码是 `admin` / `admin`，也就是示例假定的取值： + +```bash +OS_ADMIN_USER="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.username}' | base64 -d)" +OS_ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret cpaas-opensearch-admin-password \ + -o jsonpath='{.data.password}' | base64 -d)" +echo "$OS_ADMIN_USER / $OS_ADMIN_PASSWORD" +``` + +生产环境请先修改该密码，并且要和 Operator 侧保持一致，否则 Operator 自身的健康检查会失败；平台 OpenSearch 知识库里有完整改密流程（把新密码的 hash 写入 `internal_users.yml` 并生成 `securityconfig-secret`，同时创建匹配的 `admin-credentials-secret`，再更新集群）。日志账号的 `` 请填不少于 32 个字符的值。 + + +### 5.2 可选：安装中文分词插件 + +`analysis-ik` 是可选组件。不安装时使用 standard 分析器，中文不分词；安装后中文会分词。 + +请在 `OpenSearchCluster` 的 spec 中配置该插件。节点每次启动时，operator 会把 `pluginsList` 中的每一项交给 `opensearch-plugin install` 执行，因此插件在 Pod 重启、节点替换后依然存在。手工进容器安装不算数：插件写在数据卷之外，Pod 一重启就没了。 + +| 字段 | 作用 | +| --- | --- | +| `spec.general.pluginsList` | 在所有 OpenSearch 节点上安装插件 | +| `spec.bootstrap.pluginsList` | 在负责组建集群的 bootstrap Pod 上安装插件。只要是在新建集群时就要启用插件，这个字段必须一起配置，否则集群初始化可能失败。 | + +如果希望在创建集群时就带上插件，请在 apply 5.1 的 `cpaas-opensearch.yaml` 之前把这两个字段加进去： + +```yaml +spec: + general: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" + bootstrap: + pluginsList: + - "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip" +``` + +如果集群已经建好且没装插件，改用下面的 patch 命令即可，operator 会滚动重启节点完成安装： + +```bash +kubectl -n cpaas-system patch opensearchcluster cpaas-opensearch --type=merge -p '{"spec":{"general":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]},"bootstrap":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]}}}' +``` + +插件版本必须与 `spec.general.version` 一致。URL 返回 404 会导致所有节点无法启动。集群无法访问外网时，请先把 zip 放到内网服务器，并把 URL 换成内网地址。 + +校验每个节点都已安装： + +```bash +for p in $(kubectl -n cpaas-system get pod -l opster.io/opensearch-cluster=cpaas-opensearch -o jsonpath='{.items[*].metadata.name}'); do echo -n "$p: "; kubectl -n cpaas-system exec $p -c opensearch -- bin/opensearch-plugin list | grep -c '^analysis-ik'; done +``` + +每个节点都必须输出 `1`。随后确认分析器能对中文分词。请把 `` 替换为集群 Service 地址，例如 `https://cpaas-opensearch.cpaas-system.svc:9200`： + +```bash +curl -sk -u "admin:" -X POST "https:///_analyze" \ + -H 'Content-Type: application/json' \ + -d '{"analyzer":"ik_smart","text":"自然语言处理技术"}' +``` + +返回结果必须包含多个词元，例如 `自然语言`、`处理`、`技术`。 + +### 5.3 创建日志组件使用的账号 + +该账号需要读写日志索引、创建日志组件使用的索引模板，并管理其生命周期策略。security 插件已启用，因此需要通过其 REST API 创建角色、用户和角色映射。这些调用同样使用管理员账号 `admin`，密码就是 5.1 中读到的那个。 + +```bash +OS="https://" # 集群 Service，例如 https://cpaas-opensearch.cpaas-system.svc:9200 + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/roles/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{ + "cluster_permissions": [ + "cluster:monitor/*", + "cluster:admin/opendistro/ism/policy/*", + "indices:admin/index_template/put", + "indices:admin/index_template/get", + "indices:admin/template/put", + "indices:admin/template/get" + ], + "index_permissions": [{ + "index_patterns": ["log-*", "event-*", "audit-*", "meter-*"], + "allowed_actions": [ + "indices:admin/create", + "indices:admin/mapping/put", + "indices:data/write/*", + "indices:data/read/*" + ] + }] + }' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/internalusers/platform-logging" \ + -H 'Content-Type: application/json' \ + -d '{"password":"","backend_roles":[]}' + +curl -sk -u "admin:" -X PUT "$OS/_plugins/_security/api/rolesmapping/log_storage_writer" \ + -H 'Content-Type: application/json' -d '{"users":["platform-logging"]}' +``` + +模板和策略权限属于集群级权限。把 `indices:admin/index_template/put` 和 `indices:admin/template/put` 写在 `index_permissions` 下不会生效，日志组件随后会因无权限而无法创建索引模板。缺少角色映射时，账号可以认证但没有任何权限。 + +用新账号校验： + +```bash +curl -sk -u "platform-logging:" -X PUT "$OS/_index_template/perm-check" \ + -H 'Content-Type: application/json' \ + -d '{"index_patterns":["log-perm-check-*"],"template":{"settings":{"number_of_shards":1}}}' + +curl -sk -u "platform-logging:" -X POST "$OS/log-perm-check/_doc" \ + -H 'Content-Type: application/json' -d '{"check":1}' + +curl -sk -u "platform-logging:" "$OS/_index_template/perm-check" +``` + +三条命令都必须成功。出现 `security_exception` 且提示 `no permissions for [...]`，说明角色或角色映射不完整。最后用管理员账号清理： + +```bash +curl -sk -u "admin:" -X DELETE "$OS/_index_template/perm-check" +curl -sk -u "admin:" -X DELETE "$OS/log-perm-check" +``` + +连接信息请在步骤 6 记录。 + +平台下发的索引模板为 1 分片 1 副本。如果你的高可用策略需要不同的取值，请下发一个优先级更高的可组合模板，并用 `GET /_index_template` 确认结果。已创建的索引会保持创建时的设置。 + +## 步骤 6：记录连接信息 + +请记录下表各项。中间一列是安装时连接 Secret 中对应的 key。 + +### ClickHouse + +| 值 | 连接 Secret key | 从哪里获取 | +| --- | --- | --- | +| 连接地址 | `platform-default-ch-conn` → `endpoint` | 暴露 8123 端口的 cluster Service，可用 `kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse` 查询。启用 TLS 时用 `https://` 和 8443 端口 | +| Cluster 名称 | `cluster` | `ClickHouseInstallation` 的 `spec.configuration.clusters[0].name`，也是 `ON CLUSTER` 使用的名称。默认值：`replicated` | +| 数据库 | `database` | `spec.configuration.settings.default_database`。默认值：`observability` | +| 分片数 / 副本数 | 不在 Secret 中：`PlatformLogForward` 的 `externalStorage.shards` / `replicas` | `spec.configuration.clusters[0].layout.shardsCount` / `replicasCount` | +| 用户名 | `username` | `ClickHouseInstallation` 中声明的日志账号。默认值：`platform-logging` | +| 密码 | `password` | `kubectl -n cpaas-system get secret clickhouse-platform-logging-password -o jsonpath='{.data.password}' \| base64 -d` | +| CA 证书 | `tls.ca` | 仅当连接地址使用 HTTPS 且为自签 CA 时需要 | + +### OpenSearch + +| 值 | 连接 Secret key | 从哪里获取 | +| --- | --- | --- | +| 连接地址 | `platform-default-os-conn` → `endpoints` | 服务地址或负载均衡地址，例如 `https://:9200` 或 `https://cpaas-opensearch.cpaas-system.svc:9200`。请把高可用地址放在第一个 | +| 用户名 | `username` | 5.3 中创建的账号。默认值：`platform-logging` | +| 密码 | `password` | 你为该账号设置的密码 | + +### Kafka + +| 值 | 连接 Secret key | 从哪里获取 | +| --- | --- | --- | +| Bootstrap 地址 | `platform-default-mq-conn` → `bootstrap` | `:9093` 走 SASL over TLS 时为 `-kafka-bootstrap.cpaas-system.svc:9093`；不走 TLS 时用 `:9092` | +| Cluster 名称 | `kafkaClusterName` | `RdsKafka` 资源的 `metadata.name`。默认值：`cpaas-kafka` | +| 用户名 | `username` | `RdsKafkaUser` 的名称。默认值：`platform-logging` | +| 密码 | `password` | `kubectl -n cpaas-system get secret platform-logging-password -o jsonpath='{.data.password}' \| base64 -d`。在 Alauda OS 节点或其他启用 FIPS 的主机上必须至少 32 个字符 | +| Topic | `topics.log` / `topics.event` / `topics.audit` | `ALAUDA_LOG_TOPIC`、`ALAUDA_EVENT_TOPIC`、`ALAUDA_AUDIT_TOPIC` | +| SASL 机制 | `sasl_mechanism` | 除 broker 使用其他机制外，固定为 `SCRAM-SHA-512` | +| CA 证书 | `tls.ca` | 仅 9093 的 TLS 监听器需要：`kubectl -n cpaas-system get secret cpaas-kafka-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' \| base64 -d` | + +## 环境检查清单 + +### 通用 + +| 检查项 | 期望结果 | +| --- | --- | +| 节点和磁盘 | 已使用独占节点并打好标签与污点，已挂载用于持久化的 SSD，且清单中已容忍该污点 | +| StorageClass | 每个要安装的组件都有对应的 StorageClass，并能正常绑定存储卷 | +| Operator | `log-storage-operator`、ClickHouse 或 OpenSearch 对应的 Operator、Kafka Operator 均已就绪，其 CRD 已存在，且都能 watch `cpaas-system` | +| Kafka broker | 集群就绪，`message.max.bytes` 和 `replica.fetch.max.bytes` 均为 `10485760`，且 `log.retention.bytes` 与 topic 上限一致 | +| Kafka 用户和 ACL | `RdsKafkaUser` 为 `Active`，九条 ACL 均已配置 | +| Kafka topic | broker 上已存在三个 topic，且分区数、`retention.bytes` 和 `segment.bytes` 符合预期 | +| Kafka 连通性 | 已使用日志账号成功生产和消费一条记录 | + +### ClickHouse + +| 检查项 | 期望结果 | +| --- | --- | +| ClickHouse | `status.status` 为 `Completed`，所有 Pod 就绪，日志账号可以建表和删表。三节点及以上：Keeper 仲裁集群为 1 个 leader、2 个 follower。单节点：唯一的 Keeper 成员状态为 `standalone` | +| Keeper Service | 三节点及以上：4.3.1 创建的 Headless Service 能选中就绪的 Pod，且 `system.zookeeper_connection` 指向它。单节点：不创建 Service，`system.zookeeper_connection` 指向 `localhost` | +| 信息记录 | 连接地址、cluster 名称、数据库、分片数 / 副本数、用户名、密码，以及使用 HTTPS 时的 CA 证书均已记录 | + +### OpenSearch + +| 检查项 | 期望结果 | +| --- | --- | +| OpenSearch | 集群健康状态为 `green`，且日志账号可以管理索引模板 | +| 中文分词 | 启用插件时，每个节点都能看到 `analysis-ik` 且能对中文分词 | +| 信息记录 | 连接地址、用户名和密码均已记录 | + +任何一项不通过，都必须在日志组件连接该存储之前修复。 diff --git a/docs/zh/upgrade/index.mdx b/docs/zh/upgrade/index.mdx new file mode 100644 index 0000000..51190b9 --- /dev/null +++ b/docs/zh/upgrade/index.mdx @@ -0,0 +1,227 @@ +--- +weight: 15 +sourceSHA: 937bc8eb61d1deab0bedea99d3237248087d3995447ea54692289f2dbaa3a2b4 +--- + +# 升级 + +本文介绍如何升级现有 ACP 部署中的 **Alauda Container Platform Log Storage for Elasticsearch**。 + +:::warning +在本文档明确要求之前，请勿卸载 Elasticsearch 存储插件、停止旧数据链路，也不要删除其 PVC、PV 或保护 finalizer;同时请勿停止、缩容或删除旧的 Elasticsearch、Kafka、ZooKeeper、lanaya 与 Razor 工作负载。提前操作可能导致历史数据不可用，或破坏迁移边界。 +::: + +## 开始之前 + +1. 目标 ClickHouse 或 OpenSearch 集群以及新的 Kafka 服务已按[环境准备](https://docs.alauda.cn/logging-service/4.3/prepare/index.html)准备好，连接信息可用。 +2. 旧的 Elasticsearch、Kafka、ZooKeeper、lanaya 与 Razor 工作负载仍在运行，且已安排好变更窗口。 +3. 如需迁移历史数据，本版本的迁移镜像可用，并含完整的 registry、tag 或 digest。 +4. 目标端要与 `PlatformLogForward` 将要声明的取值一致。请以环境中实际部署的 CR 为准，而不是本文档： + + | 目标 | 必须对齐的项 | + | --- | --- | + | ClickHouse | `externalStorage.shards` 与 `replicas` 按 `ClickHouseInstallation`（`spec.configuration.clusters[].layout`）填写;Secret 的 `cluster` 取 `ON CLUSTER` 使用的集群名。不一致会把数据写到错误的副本集。 | + | Kafka | Secret 的 `topics.log` / `topics.event` / `topics.audit` 必须与实际 topic 名及 `KafkaUser` 的 ACL 一致，`kafkaClusterName` 与 Kafka 集群名一致;broker 与 topic 的最大消息大小必须能容纳审计批次。 | + | OpenSearch | 索引的分片与副本不由 `PlatformLogForward` 控制。平台下发的是 1 分片 1 副本的模板;如果生产需要不同取值，请在切换前下发更高优先级的模板。 | + +## 升级流程概览 + +请按以下顺序执行。每步的卡点通过后才能进入下一步。 + +| 步骤 | 执行位置 | 操作 | 继续执行的条件 | +| --- | --- | --- | --- | +| 1 | 业务集群 | 安装新数据链路：连接 Secret + 带升级 annotation 的 `PlatformLogForward` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | +| 2 | 业务集群 | 如需历史数据，按[历史数据迁移](https://docs.alauda.cn/logging-service/4.3/migration/index.html)创建 `LegacyESMigration` | `PrecaptureReady`（已有迁移时为 `Succeeded`） | +| 3 | global 集群与业务集群 | 执行旧插件的受控卸载 | 60 秒内 `ModuleInfo`、`ClusterPluginInstance`、`AppRelease` 均未出现，且新日志查询正常 | +| 4 | 业务集群 | 观察同一个 `LegacyESMigration` | `Phase=Succeeded` 且目标端查询通过 | +| 5 | 业务集群 | 保留受保护的源端存储卷 | 清理前需获得明确批准 | + +## 升级步骤 + +### 步骤 1：带着升级 annotation 安装新数据链路 + +新数据链路的安装方式与全新安装完全相同：先创建目标存储和 Kafka 的连接 Secret，再创建一个 `PlatformLogForward`。相关清单与说明见[安装](https://docs.alauda.cn/logging-service/4.3/install/index.html)中的「外部 ClickHouse 或 OpenSearch 的日志存储安装」章节。 + +**唯一的区别是 annotation。** 请在 `PlatformLogForward` 上添加 `log.alauda.io/legacy-es-upgrade: "true"`，让平台进入旧 Elasticsearch 升级流程，而不是直接切换日志入口： + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default # 集群内固定的单例名称，请勿修改 + annotations: + log.alauda.io/legacy-es-upgrade: "true" # 固定值，进入旧 Elasticsearch 升级流程 +spec: + installMode: Fresh # 始终为 Fresh，请勿改为 Adopt + # externalStorage 与 externalMessageQueue：同「安装」章节， + # 指向上面创建的目标存储与 Kafka 连接 Secret +``` + +请保持 `installMode: Fresh`，`Adopt` 不属于本流程。 + +新数据链路不会立即就绪。平台会先创建链路，然后切换日志入口，最后等待旧集群中排队的数据被消费完；耗时取决于积压量。请观察状态直到完成，按 `Ctrl+C` 结束： + +```bash +kubectl get platformlogforward platform-default -w +``` + +`Phase` 列变为 `Ready`，同时 `Ready` 列变为 `True`。只有看到 `Ready` 后才能继续。 + +如需跟踪进度或排查问题，可以查看状态 conditions： + +```bash +kubectl get platformlogforward platform-default \ + -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' +``` + +请关注 `LegacyESUpgrade` 这一行：当 reason 变为 `LegacyESUpgradeCompleted` 时，说明日志入口已切换到新数据链路，且旧集群中排队的数据已消费完，本步骤完成。如果 reason 为 `Blocked`，`message` 会说明原因。 + +本步骤完成后，请生成或找到新的日志、事件和审计记录，确认可以从新目标端查询到，然后再继续。`LegacyESUpgrade` 未完成前，请勿卸载旧插件。 + +如果源端集群使用旧 Kafka，请勿停止或缩容 Kafka、ZooKeeper 或 lanaya。平台会自动通过旧链路排空排队数据，并在旧消费组 lag 归零时记录 `LegacyKafkaDrained`；`LegacyESUpgradeCompleted` 是本流程的卡点。 + +### 步骤 2：如需历史数据，创建迁移资源 + +只有在已批准的升级方案确认**不需要**历史 Elasticsearch 数据时才可跳过本步骤，并请记录该决定。不要为了跳过迁移而删除源端 PVC 或 PV。 + +如果需要这些数据，请在卸载旧插件**之前**创建 `LegacyESMigration`。提前创建才会记录最终源状态与源端存储卷;卸载之后再创建，必须显式指定源卷，无法使用最终快照。 + +迁移的支持范围、迁移时间范围必须落在目标保留期内的要求、清单与校验步骤，全部见[历史数据迁移](https://docs.alauda.cn/logging-service/4.3/migration/index.html)。 + +### 步骤 3：卸载旧 Elasticsearch 存储插件 + +:::warning +只有在以下条件全部满足时才能执行本步骤： + +- `PlatformLogForward/platform-default` 为 `Ready`，且其 `LegacyESUpgrade` condition 的 reason 为 `LegacyESUpgradeCompleted`。 +- 如需历史迁移，`LegacyESMigration/platform-es-history` 为 `PrecaptureReady` 或 `Succeeded`。 +- 请确认升级程序已完成旧 Elasticsearch 存储卷（PVC/PV）的保护。请勿修改或移除该保护；如果平台报告保护失败，请停止并联系支持。 +- 如果源端使用旧 Kafka，平台已确认所有排队数据均已排空。请勿通过停止或缩容 Kafka、ZooKeeper、lanaya 或 Elasticsearch 来强制达到该状态。 +::: + +:::warning +本流程会临时清理 Alauda Container Platform Log Storage for Elasticsearch 的平台发现字段，以便卸载旧插件。请仅在批准的变更窗口内执行，并请勿修改或删除 logagent 及其依赖。 +::: + +如果平台提供了本次升级支持的插件卸载操作，请先按平台或支持人员的说明执行。如果该操作被拒绝，或平台团队要求执行受控流程，请使用下面的步骤。 + +两次删除必须连续执行：中间不要等待 `ModuleInfo` 消失、不要检查 `AppRelease`，也不要执行其他检查。 + +**在 global 集群：开始关键步骤** + +将 `CLUSTER` 设置为业务集群在 global 集群中注册的名称，然后执行： + +```bash +set -euo pipefail + +CLUSTER= + +# 1. 解析目标 ModuleInfo。结果不唯一时停止。 +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')" +if [ "$MODULE_COUNT" -gt 1 ]; then + echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2 + exit 1 +fi + +if [ "$MODULE_COUNT" -eq 1 ]; then + MODULE_INFO="$MODULE_INFOS" + MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')" + MODULE_CONFIG="logcenter-${MODULE_VERSION}" + + # 2. 修改前先备份当前的管理对象。 + kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml + kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml + kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml + + # 3. 清理发现标记，这一步同时绕过 logagent 的依赖检查。 + kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}' + kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}' + + # 4. 删除请求被接受后立即返回，不要在此等待。 + kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false +else + echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record." +fi +``` + +**在业务集群：完成关键步骤** + +立即把 kubectl context 切换到业务集群，不要先执行任何等待或校验。执行： + +```bash +# 5. 删除单集群安装记录，避免其重新创建 ModuleInfo。 +kubectl delete clusterplugininstance logcenter --ignore-not-found +``` + +**等待旧数据链路移除** + +切回 global 集群，等待 `ModuleInfo` 消失： + +```bash +CLUSTER= +MODULE_INFOS="$(kubectl get moduleplugin logcenter \ + -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')" +MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')" + +if [ -n "$MODULE_INFO" ]; then + kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m +else + echo "No ModuleInfo remains for cluster $CLUSTER." +fi +``` + +切换到业务集群，等待旧 `AppRelease` 被删除： + +```bash +kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m +``` + +完成这些步骤后，等待 60 秒并确认这些资源没有重新出现。每条命令都不应返回资源。 + +**在 global 集群** + +```bash +kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=' --ignore-not-found +``` + +**在业务集群** + +```bash +kubectl get clusterplugininstance logcenter --ignore-not-found +kubectl -n cpaas-system get apprelease logcenter --ignore-not-found +kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found +``` + +如果 `ClusterPluginInstance/logcenter` 仍然存在或重新出现，请再次删除并重复检查；只要它存在，平台就可能重新创建 `ModuleInfo`。如果 `ModuleInfo` 也重新出现，请先删除 `ClusterPluginInstance/logcenter`，再删除新的 `ModuleInfo`。请勿手动恢复被清理的发现字段；只需确认 `ModuleInfo`、`ClusterPluginInstance` 和 `AppRelease` 不再出现。如果 global 集群的步骤在删除 `ModuleInfo` 之前失败，请勿执行本次的业务集群步骤：先修复原因，再重新执行 global 集群步骤。 + +稳定性检查通过后，生成新的日志、事件和审计记录，并确认可以从新目标端查询到。如果新数据链路不健康，请勿继续迁移校验。 + +### 步骤 4：保留的存储卷 + +受保护的旧 Elasticsearch PVC 和 PV 会被保留。请在迁移和目标端校验完成前保留它们。请勿移除保护注解、删除 PVC/PV 或移除 finalizer。校验完成后如需释放这些存储卷，请联系 Alauda 支持或按照单独的、已批准的清理流程执行。 + +本次升级只保留 ES 的 PVC 和 PV。旧 Kafka 和 ZooKeeper 的存储卷不在保留范围内。 + +请导出或记录最终的 PLF conditions、迁移状态、目标端查询结果以及保留的 PVC/PV 名称，用于实施交接。请勿在本次升级中移除保护注解或删除这些存储卷。 + +## 完成检查清单 + +| 状态 | 期望 | +| --- | --- | +| `PlatformLogForward/platform-default` | `Phase=Ready` 且 `LegacyESUpgradeCompleted` | +| `LegacyESMigration/platform-es-history`（如已创建） | `Phase=Succeeded` | +| 新日志查询 | 新的日志、事件和审计记录可以通过新查询链路返回 | +| 旧插件卸载 | `ModuleInfo`、`ClusterPluginInstance/logcenter`、`AppRelease/logcenter` 均不存在，旧 ES 工作负载已消失 | +| 源端存储卷 | 受保护的旧 ES PVC 和 PV 仍然存在 | + +如果任何状态不符合预期，请停止并联系 Alauda 支持。请勿通过删除迁移资源、源端存储卷或目标端数据来绕过故障。 + +## 步骤受阻时 + +- 平台以 `moduleinfo is depended by ...` 拒绝卸载插件：请停止，不要修改 logagent，重新执行受控流程；如果仍然失败，请联系支持。 +- 迁移为 `Blocked`、`Failed` 或一直停留在 `Running`：请勿删除迁移资源、Job、源端存储卷或目标端数据；请联系支持。 +- `ModuleInfo` 或 `ClusterPluginInstance` 重新出现：请先删除 `ClusterPluginInstance`，再删除新的 `ModuleInfo`，并重复 60 秒稳定性检查。 +- `AppRelease/logcenter` 未消失：请勿移除 finalizer；请联系支持。 +- 卸载后新日志查询失败：请停止并联系支持。请勿删除保留的存储卷。