Skip to content

MIGRATION-RUNBOOK.md

Latest commit

 

History

History
275 lines (190 loc) · 10.5 KB

MIGRATION-RUNBOOK.md

File metadata and controls

275 lines (190 loc) · 10.5 KB
Raw

CAPMOX Migration Runbook: v0.7 → v0.8 (v1alpha1 → v1alpha2)

Overview

Component Old Version New Version
Cluster API (CAPI) v1.10 (v1beta1) v1.12 (v1beta2)
CAPMOX v0.7.x (v1alpha1) v0.8.x (v1alpha2)
Kubernetes (workloads) ≤ 1.31 v1.35.3
CNI — Cilium (kube-proxy replacement, CRS-installed)

Why not clusterctl upgrade apply? i encoutered issues when migrating it.

Why not clusterctl move from the old mgmt? Two blockers:

  1. clusterctl v1.12 refuses a v1beta1 source:

    "this version of clusterctl could be used only with 'v1beta2' management clusters, 'v1beta1' detected"

  2. CAPI maintainers (kubernetes-sigs/cluster-api#13034) state plainly: "clusterctl move has not been designed to handle migration nor upgrade paths."

So the supported path is: new management cluster + recreate workload clusters on it + cut traffic over + delete on the old side. It's more manual, but it's the only option that actually works for a v1beta1 → v1beta2 fleet.

Are my workload VMs safe? Yes — until you decide to retire each one. The old v0.7 mgmt keeps its workload clusters alive and paused. You migrate them one at a time on your own schedule.

Prerequisites

Install these tools on your workstation:

  • kubectl (v1.35+)
  • clusterctl (v1.12+) — install guide
  • kind (for the bootstrap cluster)
  • helm (for Cilium CNI rendering)
  • docker (for Kind)

Step-by-Step Migration

Phase 1: Prepare

  1. Edit clusterctl.yaml with your Proxmox URL, token, and infrastructure details. Copy it to ~/.cluster-api/clusterctl.yaml.

  2. Edit manifests/01-proxmox-credentials.yaml with your Proxmox API credentials.

  3. Edit manifests/10-future-mgmt-cluster.yaml — replace:

    • <MGMT_VIP_ADDRESS> — the VIP for the new management cluster's kube-apiserver
    • IP range, gateway, DNS servers for your subnet
    • sourceNode, templateID for your K8s 1.35.3 VM template

  4. Build a K8s v1.35.3 VM template on Proxmox using image-builder. See skills/proxmox-k8s-template/ for a walkthrough.

Phase 2: Backup + pause (on OLD management cluster)

chmod +x scripts/*.sh
./scripts/01-backup-existing.sh

This will:

  • Discover all workload clusters
  • Pause them (spec.paused: true — stops reconciliation but not traffic)
  • Export every CAPI/CAPMOX resource to YAML
  • Create a portable clusterctl move --to-directory backup

Verify the backup directory is populated before proceeding.

Phase 3: Kind bootstrap + v0.8 providers

./scripts/02-setup-new-mgmt.sh

This will:

  • Create a Kind cluster (capmox-mgmt-v08) as a temporary bootstrap
  • Install CAPI v1.12 + CAPMOX v0.8 + in-cluster IPAM
  • Enable EXP_CLUSTER_RESOURCE_SET=true for Cilium auto-install
  • Wait for all controllers to become ready

Then apply credentials on Kind:

kubectl --context kind-capmox-mgmt-v08 \
  apply -f manifests/01-proxmox-credentials.yaml

Phase 4: Provision the future Proxmox-hosted mgmt

./scripts/03-provision-new-mgmt.sh

This will, from Kind:

  1. Render Cilium → ConfigMap + apply manifests/20-cilium-crs.yaml
  2. Apply manifests/10-future-mgmt-cluster.yaml
  3. Wait for the kubeconfig Secret, retrieve it to ~/.kube/capmox-mgmt-prod.kubeconfig
  4. Wait for Cluster.status.conditions[Available]=True (v1beta2 — rolls up CP + workers + infra)
  5. Run clusterctl init ON the new Proxmox-hosted mgmt (target of the pivot must have providers installed)
  6. Apply the credentials Secret on the new mgmt

Phase 5: Pivot (Kind → Proxmox mgmt)

./scripts/04-pivot-to-proxmox.sh

This does clusterctl move from Kind into the Proxmox-hosted mgmt, verifies the move, then deletes Kind. At this point your new mgmt is self-managed on Proxmox.

Note on naming: this is the "pivot" phase, separate from the workload-cluster migration below. Because the pivot is Kind → new-mgmt (both v1beta2), it works. The problem begins when you try to move v0.7 → v0.8.

Phase 6: Migrate old workload clusters — MANUAL, per cluster

There is no script for this. Here's why, and how to do it.

Why no script

clusterctl move from the old v0.7 (v1beta1) mgmt into the new v0.8 (v1beta2) mgmt is blocked at multiple layers:

Blocker Detail
clusterctl v1.12 Refuses a v1beta1 source cluster (hard error)
CAPMOX v0.8.0 v1alpha1 → v1alpha2 conversion webhook is broken
CAPI issue #13034 Maintainers: "clusterctl move has not been designed to handle migration nor upgrade paths"

Recreation on the new mgmt is the only path. It also means new VMs — so you run each workload in parallel on old + new, cut over when the new side is healthy, then delete on the old side.

Per-cluster procedure

For each workload cluster still managed by the old v0.7 mgmt:

OLD_CTX="capi-host-admin@capi-host"
NEW_KCONF="${HOME}/.kube/capmox-mgmt-prod.kubeconfig"
NAME="<cluster-name>"
NS="<namespace>"

1. Back up app workloads running inside the cluster (Velero, PVC snapshots, kubectl get -A -o yaml, or whatever your stack expects). CAPMOX does not know about your apps.

2. Create the replacement manifest on the new mgmt. Copy the template:

cp manifests/02-workload-cluster.yaml manifests/02-workload-${NAME}.yaml
# Then edit:
#   - metadata.name everywhere (match the old cluster so DNS/ingress swap is cleaner)
#   - spec.controlPlaneEndpoint.host (unique VIP per cluster)
#   - InClusterIPPool addresses (unique per cluster)
#   - sourceNode, templateID, replicas, vCPU/memory

3. Apply on the NEW mgmt:

kubectl --kubeconfig "${NEW_KCONF}" \
  apply -f manifests/02-workload-${NAME}.yaml

4. Wait for Available=true (v1beta2 condition — rolls up control plane, workers, infrastructure):

kubectl --kubeconfig "${NEW_KCONF}" \
  wait cluster "${NAME}" -n "${NS}" \
  --for=condition=Available --timeout=15m

5. Restore app workloads into the new cluster.

6. Cut over DNS / ingress / clients to the new cluster's VIP. Verify traffic is landing on the new nodes.

7. Unpause the old cluster (Phase 2 paused it), then delete it:

kubectl --context "${OLD_CTX}" \
  patch cluster "${NAME}" -n "${NS}" \
  --type merge -p '{"spec":{"paused": false}}'

kubectl --context "${OLD_CTX}" \
  delete cluster "${NAME}" -n "${NS}"

Gotcha: if you skip the unpause, the delete sits in Terminating forever. A paused Cluster's controllers don't reconcile, so finalizers never run and CAPMOX v0.7 never drains the VMs. Always unpause before you delete.

8. Watch the old cluster drain:

watch kubectl --context "${OLD_CTX}" \
  get machines,proxmoxmachines,cluster -A

Once the Cluster object is gone and CAPMOX has reaped the VMs, the old cluster is done. Repeat for the next workload cluster.

Phase 7: Decommission the old v0.7 management cluster

Only after the old mgmt is empty:

kubectl --context capi-host-admin@capi-host get clusters -A
# No resources found.

Then shut it down by whatever means you used to bring it up (Kind delete, VM delete, etc.).

Phase 8: Deploy brand-new clusters on K8s v1.35.3

For brand-new clusters (not migrations), copy the workload template, edit it, and apply on the new mgmt:

kubectl --kubeconfig "${NEW_KCONF}" \
  apply -f manifests/02-workload-cluster.yaml

Cilium installs automatically via the ClusterResourceSet (manifests/20-cilium-crs.yaml) as soon as ControlPlaneInitialized flips True. If you'd rather install manually, see the HelmChartProxy / helm install alternatives in manifests/03-cilium-cni.yaml.

Phase 9: Validate

# Check all clusters on the new mgmt
kubectl --kubeconfig "${NEW_KCONF}" get clusters -A

# Check all machines are Running
kubectl --kubeconfig "${NEW_KCONF}" get machines -A

# Get a workload kubeconfig
KUBECONFIG="${NEW_KCONF}" \
  clusterctl get kubeconfig <cluster-name> > /tmp/workload.kubeconfig

# Verify workload nodes
kubectl --kubeconfig /tmp/workload.kubeconfig get nodes -o wide

# Verify Cilium is healthy
kubectl --kubeconfig /tmp/workload.kubeconfig -n kube-system \
  get pods -l app.kubernetes.io/name=cilium

Key Changes: v1alpha1 → v1alpha2

What Changed v1alpha1 v1alpha2
API version infrastructure.cluster.x-k8s.io/v1alpha1 infrastructure.cluster.x-k8s.io/v1alpha2
Credentials Environment variables / clusterctl.yaml credentialsRef pointing to a Secret
Network config Per-field in ProxmoxCluster spec Simplified ipv4Config block
Credentials Secret Not required Required with label platform.ionos.com/secret-type: proxmox-credentials
Top-level CAPI condition ControlPlaneReady (v1beta1) Available (v1beta2, rolls up CP + workers + infra)
Cilium install manual helm install ClusterResourceSet via EXP_CLUSTER_RESOURCE_SET=true

Troubleshooting

kubectl delete cluster sits in Terminating forever on the old mgmt: The cluster is still paused from Phase 2. Unpause first:

kubectl --context capi-host-admin@capi-host \
  patch cluster <name> -n <ns> --type merge \
  -p '{"spec":{"paused": false}}'

CAPMOX v0.7 will then wake up and drain the finalizers.

Cluster stuck in Provisioning after apply on new mgmt: Check that the proxmox-credentials Secret exists in the correct namespace, has the platform.ionos.com/secret-type: proxmox-credentials label, and the credentialsRef name in ProxmoxCluster matches.

Machines not becoming Ready on the new mgmt: Verify the VM template ID exists on the specified sourceNode and the IP pool has available addresses. Also check that the VIP is free on the network (not occupied by another kube-vip instance).

clusterctl refuses to read the old mgmt cluster: This is expected. See the "Why no script" section above. Recreate, don't move.

Cilium pods CrashLooping on a new cluster: Confirm kubeProxyReplacement: true is set in the CRS-rendered Cilium ConfigMap and k8sServiceHost points to the cluster's own VIP, not the mgmt VIP.

CCM / cloud-provider issues: This migration intentionally does not deploy the Proxmox cloud-provider-external. kube-vip handles the control-plane VIP; kubelet populates providerID directly via cloud-init ({{ ds.meta_data.instance_id }} resolves to the SMBIOS UUID). If you need a CCM later, add it after the cluster is Available=True.