Conversation
Add docs/en/upgrade/: an Upgrade section entry page and the guide for upgrading a cluster that stores logs with Log Storage for Elasticsearch to ACP 4.4, where new data is written to ClickHouse or an OpenSearch 3.7.0 cluster. The guide covers: - connection Secrets for the target storage and the new Kafka service - the new data path created with a PlatformLogForward that carries the log.alauda.io/legacy-es-upgrade annotation - controlled uninstall of the Log Storage for Elasticsearch plugin - optional migration of the retained historical data - cleanup of the retained Elasticsearch PVCs and PVs
Deploying alauda-logging-service with
|
| Latest commit: |
e950980
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://5b9829d4.alauda-logging-service.pages.dev |
| Branch Preview URL: | https://docs-elasticsearch-upgrade-g.alauda-logging-service.pages.dev |
chinameok
requested changes
Sep 11, 2026
chinameok
left a comment
chinameok
left a comment
Contributor
There was a problem hiding this comment.
不需要新增一个 elasticsearch-upgrade.mdx 文件，写在 docs/en/upgrade/index.mdx 更方便。
Author
|
已按 review 意见调整：升级指南已合并进 |
Author
|
已在测试环境完成一次 ClickHouse 目标端到端验证：legacy ES 3 节点 + 旧 Kafka/ZK 已存在；安装 4.4 log-storage-operator / ClickHouse / Strimzi 后，PLF 达到 |
added 18 commits
September 14, 2026 18:36
This reverts commit 7912800.
- add target storage preparation with capacity-planning reference and preflight checklist - align target settings with the deployed operator CRs (ClickHouseInstallation, Kafka/KafkaNodePool/KafkaTopic, OpenSearch templates) - treat analysis-ik as optional and base acceptance on resource status - keep product-exposed options, index examples, and migration status details
Translate docs/en/upgrade/index.mdx to Simplified Chinese under docs/zh, with sourceSHA for the translation pipeline. Code blocks stay identical; only comments are localized.
This reverts commit 9f55448.
Add docs/en/prepare/index.mdx, covering the ClickHouse or OpenSearch and Kafka backends the logging components connect to when they do not create the storage themselves. The guide was validated on an ACP 4.3 cluster, and the following points come from that run: - Every operator must watch cpaas-system, where the storage and messaging resources are created. A namespace-scoped OperatorGroup makes them ignore those resources silently, without status, events, or pods. - The ClickHouseInstallation must not set fsGroup/runAsUser 101, and the logging account name needs backticks because it contains a hyphen. - The Keeper is mandatory for every profile. A three-node quorum uses fully qualified pod names for its raft peers; short pod names do not resolve and the Keeper then never opens its client port. - The RdsKafka controller needs roles: [controller], and its settings are static broker configuration, so they are read from the broker config rather than from kafka-configs.sh. - The RdsKafkaUser readiness condition is status.phase: Active. - The OpenSearch role needs the template permissions at cluster level, plus a role mapping, and analysis-ik is installed through the spec pluginsList fields so that it survives a pod restart.
Rewrite the storage, Kafka, and OpenSearch steps so that an implementer can follow the chapter without filling in steps themselves: - Every YAML block now names the file to save it as and the kubectl apply command to run; all commands run from a host with kubectl access. - The ClickHouse section ships two complete manifests, single node and three nodes and above, instead of one manifest plus fragments that had to be merged by hand, and the three-node Keeper quorum has a manifest of its own with three explicit commands to confirm the leader and followers. - The logging account step runs the SQL inside the pod, reading the admin password from the Secret and generating the account password, instead of leaving the reader to work out where to run the SQL. - Static local volumes get a StorageClass and PV example plus the number of PVs each profile needs. - The analyzer plugin is verified with a single-line command and can be enabled on an existing cluster with one patch. Fix contradictions and gaps found while reviewing: - Step 1 taints the infra nodes, but no manifest tolerated that taint, so the storage pods could not land on those nodes. Every manifest now selects node-role.kubernetes.io/infra and tolerates it; the Kafka CRD has no nodeSelector field, so it uses nodeAffinity instead. Verified against the StatefulSets and node pools the three operators generate. - The retention period for Kubernetes logs is 30 days, not 7. - The Kafka controllers were missing from the capacity table, the Kafka CA certificate had no source, and the OpenSearch CA now points at the Secret the operator creates. - Drop the claim about who creates the observability database and the instruction to confirm name resolution from another cluster. Validated on ACP 4.3: 39 command blocks pass bash -n, all manifests pass server-side dry run, and the ClickHouse single node and three node paths, Kafka, and OpenSearch were deployed and exercised end to end.
The ClickHouse operator in the platform ships only ClickHouseInstallation, ClickHouseInstallationTemplate and ClickHouseOperatorConfiguration. There is no Keeper custom resource, and a ClickHouseInstallation cannot declare a Keeper quorum either: its schema has no Keeper field, and a raft_configuration placed under configuration.settings is dropped, leaving every pod with the same server_id. Deploy the Keeper the way the Keeper-in-ClickHouseInstallation procedure does instead, so that everything stays operator-managed and no raw workload is created: - 3.2 now only creates the headless Keeper client Service. - The three-node manifest keeps the static Keeper configuration in the cluster files, pulls in a generated file with include_from, and lets an init container write the per-pod server_id and raft member list. - The readiness probe checks the Raft port. This is required: pod creation is serialized, and with the default HTTP probe the first replica can never form a quorum, so ClickHouse never reports ready and the remaining replicas are never created. - A wait-for-self-dns init container guards the distributed DDL worker against a pod that starts before its headless Service DNS record exists. Also fixed while testing: the three-node manifest was missing the users block, so the admin account did not exist and step 3.5 could not work, and it defined no service template, which made the operator create a LoadBalancer Service whose external address stays pending. Verified on a clean deployment: the installation reaches Completed, the Keeper reports one leader and two followers, system.zookeeper_connection points at the Keeper Service, CREATE TABLE ON CLUSTER reaches all three replicas, a ReplicatedMergeTree row written on one replica is readable on all three, and the account steps run as written.
added 12 commits
September 16, 2026 23:25
Reviewing the ClickHouse steps against the Keeper-in-ClickHouseInstallation procedure turned up two gaps. The Keeper path sat at /var/lib/clickhouse-keeper while the data volume is mounted at /var/lib/clickhouse, so the Keeper log and snapshots landed in the container filesystem and were lost on every pod restart. The procedure itself asks for that path to be on the same persistent volume, which is also what the operator's own fixture does, so the path now sits under the mount as /var/lib/clickhouse/coordination. Verified by writing a marker inside the pod and reading it back from the local volume on the node. The manifest also omitted the self-observability system table TTLs that the procedure sets. Those tables grow without bound and eventually fill the data volume, so asynchronous_metric_log, metric_log and trace_log now carry the 7-day delete TTL on both the single-node and the replicated manifests. Redeployed from scratch and re-verified: the installation reaches Completed, Keeper reports one leader and two followers, CREATE TABLE ON CLUSTER reaches all three replicas, a replicated row written on one replica is readable on all three, and the account steps run as written.
Compared this chapter with the OpenSearch and Kafka articles in the platform knowledge base and fixed what they cover and we did not. OpenSearch: - The account steps authenticate as `admin` but never said where that password comes from. State that the operator stores it in the `<cluster-name>-admin-password` Secret, that `admin` / `admin` is the default the examples assume, give the command that reads it, and warn that the password has to be changed together with the operator or the operator's own health checks break. - Note that OpenSearch needs `vm.max_map_count` at 262144, that the operator sets it from an init container, and that only Pod Security Admission restricted namespaces need it set on the nodes by hand. Kafka: - Add `auto.leader.rebalance.enable: "false"`, which the knowledge base lists as a recommended default so partition leaders are not moved without an operator deciding to. - State that the operator applies hard pod anti-affinity and add the command that shows the three brokers on three distinct nodes.
Reviewing the previous commit turned up three problems it introduced. The admin credential explanation was placed in 5.3, but 5.2 already uses `<admin-password>` in its analyze call, so the reader met the placeholder before the paragraph that explains it, and the paragraph claimed the placeholder appears "below" while it also appeared above. Move the explanation to the end of 5.1, where the cluster and its generated Secret exist, and point 5.3 back at it. Reading the credentials used a single jsonpath expression that concatenates two fields and pipes them through a read loop. Replace it with the two separate jsonpath reads already used for the ClickHouse admin password in 3.5, which is the form that has been exercised. Two wording fixes: the moved paragraph opened with "These calls" while nothing preceded it in 5.1, and the Kafka anti-affinity note ended with a sentence that said the same three nodes are the minimum to schedule rather than the actual constraint, which is that hard anti-affinity needs three schedulable nodes.
Two corrections against the version-matched knowledge base. Drop `auto.leader.rebalance.enable`. It came from the Kafka best practices article, which is written for ACP 3.14/3.15, and that article itself says to verify defaults against the operator actually installed. Nothing in the 4.x material asks for it. Rework the node-local storage steps around the ACP 4.3 procedure for Kafka on local disks, which lists the field failures this chapter was open to: a pod coming up with an empty or foreign log directory, and pods binding to each other's disks after an instance is recreated. The fix it prescribes is to reserve every PersistentVolume for one specific claim with `spec.claimRef`, and that is now what the chapter does: - Step 1 reserves each volume, adds the `project.cpaas.io/ALL_ALL` label the pvc-validator webhook requires in a project namespace, sets `allowVolumeExpansion: false`, and notes that `capacity.storage` is matching metadata rather than a quota. - Step 1 lists the claim-name shape per component, and gives Kafka its own directories per pod so the paths match the volumes that reference them. - Kafka's broker claim names contain a hash generated per instance, so the claim names cannot be known in advance. Creating the instance and then reserving the volumes for the claims it produced is now part of 4.2, before the wait-and-verify step. Verified on the cluster: the six Kafka claims went from Pending to bound one-to-one with their reserved volumes and all six pods started, and a three-node OpenSearch pool came up green with volumes reserved the same way.
…apter Installing the log storage against a storage or Kafka service you provide belongs in the installation chapter, not in the upgrade flow. The upgrade chapter was carrying the entire manifest set for it: the two connection Secrets, the PlatformLogForward with all its comments, and the notes about shard and replica counts. Add that procedure to the installation chapter as Install Log Storage with External ClickHouse or OpenSearch, covering the OpenSearch and ClickHouse Secret variants, both PlatformLogForward variants, and the readiness check. The upgrade chapter now links to it and documents only what makes the two different: the log.alauda.io/legacy-es-upgrade annotation and the path it selects, which waits for the queued legacy data instead of switching the log entry point directly. Its steps renumber from six to five, and the flow table matches the new numbering.
…lickHouse account Move the external storage install procedure out of install_log.mdx into a new Installation chapter. The older page covers installing the platform storage plugins and is not where this path belongs; the new page takes the procedure that the upgrade chapter used to carry, without the upgrade annotation. Create the ClickHouse account the way the product always has. Running the whole flow end to end showed the SQL user the chapter created needed four grants it was never given: SELECT on system.clusters for razor's readiness probe, SELECT on system.zookeeper for the keeper probe, CLUSTER for ON CLUSTER DDL, and TABLE ENGINE ON Distributed for the distributed table the migrations create. The product's own ClickHouse chart instead declares the account in the ClickHouseInstallation users section with GRANT ALL ON *.* WITH GRANT OPTION, which carries all of those without a grant statement anywhere. This chapter now does the same: 3.1 creates the two password Secrets, the manifests declare admin and platform-logging, and 3.5 just reads the password and verifies it. Verified end to end against ClickHouse and Kafka prepared by this chapter: the account runs all four operations with no grants at all, the PlatformLogForward reaches Ready, schema-migrations completes, and 19 tables exist in observability.
Mirror the two chapters that were just reworked, so the Chinese tree matches the English one: - docs/zh/install/index.mdx, the new external storage installation chapter, without the upgrade annotation and with the same OpenSearch Secret shape as the English page. - docs/zh/prepare/index.mdx, including the ClickHouse account declared in the ClickHouseInstallation with GRANT ALL ON *.* instead of a SQL user with grant statements. Checked both pairs: the code blocks are identical after stripping comments, the heading ranks match, every bash block passes bash -n, every YAML block parses, and each sourceSHA matches its English source.
The upgrade chapter was carrying the whole historical data migration: the LegacyESMigration manifests for both targets, the capture and copy sequence, the batch and concurrency options, and the verification. That made the chapter read as two procedures in one. Move it to its own chapter, Historical Data Migration. The upgrade chapter keeps the upgrade itself: install the new data path with the annotation, create the migration resource when history is required, uninstall the legacy plugin, and keep the protected volumes. Where migration applies it links to the new chapter instead of repeating it. The new chapter also states the supported scope, which the upgrade chapter never did: - the source is the legacy Elasticsearch plugin's PVCs, discovered automatically while the legacy StatefulSet is present or listed explicitly afterwards; - the data is the indices selected by indexScope, which must match at least one index; - the target is ClickHouse or OpenSearch, using the same connection Secret as the PlatformLogForward; - the boundary is captured when the resource is created and the copy runs after the uninstall; - ClickHouse runs one job at a time, OpenSearch accepts two; - and the migrated time range has to fit inside the target retention, because the migration does not widen or bypass the TTL that PlatformLogForward and razor already applied.
…pter Mirror the English split: the Chinese upgrade chapter now keeps only the upgrade steps and links to a new Historical Data Migration chapter that carries the LegacyESMigration manifests, the capture and copy sequence and the verification. Also add a warning to both migration chapters: the source volumes have to be provided in time. Discovery only works while the legacy StatefulSet and its PVCs are still present, and the operator never guesses a source volume, so the PVCs must be listed in source.pvcRefs, or the resource created, before anything deletes the claims or releases the protected volumes. Checked both pairs: identical code blocks after stripping comments, matching heading ranks, and each sourceSHA matching its English source.
Both chapters opened with sections that repeated each other and the procedure: an introduction that narrated the steps, a scenarios table, a target preparation section, an alignment table and a preflight checklist all restated the same preconditions before the reader reached the first actionable step. Keep what the reader cannot infer and drop the rest: - Upgrade keeps one short introduction, one warning about not touching the legacy workloads or volumes, a four-item Before you start that points at the preparation chapter, one compact alignment table (the only place the shards, topics and index-template constraints are stated), the flow table and the steps. - Migration keeps the supported scope table and the source volume warning, and its Before you start shrinks to the two facts that matter: the new data path has to be ready with the migration image available, and the migrated range has to fit inside the target retention. English and Chinese are trimmed the same way; both pairs keep identical code blocks, matching heading ranks and matching sourceSHA values.
- prepare: split ClickHouse and OpenSearch, single-node and multi-node, and make every step runnable in the field - prepare: add log-storage-operator, and cap Kafka disk with retention.bytes / log.retention.bytes - install: add the Log Essentials and Log Collector cluster plugins - add the Chinese translations
The logging operator's auto-migration treats ClickHouseInstallation/cpaas-clickhouse, RdsKafka/cpaas-kafka, and Secret/clickhouse-basic-auth as a chart-installed deployment. Those are the names Environment Preparation creates, so an operator start after preparation makes it create PlatformLogForward/platform-default with installMode=Adopt. installMode is immutable, so the Installation manifest is then rejected with "installMode is immutable". - install: check for an existing PlatformLogForward before applying, and document the delete-and-apply recovery plus the connection Secret ownership annotation - prepare: install the operators before creating the storage resources
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An Upgrade guide for moving a cluster that stores logs with Alauda Container Platform Log Storage for Elasticsearch to ACP 4.4, where new data is written to ClickHouse or to an OpenSearch 3.7.0 cluster.
The guide is implemented directly in
docs/en/upgrade/index.mdx; no separate page is added. The separateelasticsearch-upgrade.mdxpage was removed per review feedback.What the guide covers
PlatformLogForwardthat carries thelog.alauda.io/legacy-es-upgradeannotation.LegacyESMigration, created before the controlled uninstall and observed throughPrecaptureReadytoSucceeded.Supported source Logging plugin versions:
4.2.x,4.3.x.Verification
yarn lint— 0 errors, 0 warnings.yarn build— passes;/upgrade/renders the merged guide.bash -n.