CKS cluster setup hangs in guest waiting for /mnt/k8sdisk binaries (CloudStack 4.22 + RHEL 9.7 KVM + Ubuntu CKS template)

[rishabhjain1997]

Summary

CKS cluster provisioning consistently fails on our environment. The control VM boots, cloud-init runs to completion with DataSourceCloudStack, but the in-VM setup script then spends ~25 minutes polling for /mnt/k8sdisk/ to be populated by the Kubernetes binaries ISO, gives up, and kubelet.service never gets installed. CKS then reports Failed to setup Kubernetes cluster : <name> is not in usable state as the system is unable to access control node VMs of the cluster, which is a downstream symptom — the real failure is that the binaries ISO contents never appear inside the guest.

We've reproduced this with both a custom-patched Ubuntu cloud-image and the canonical ShapeBlue/Apache CKS template (cks-ubuntu-2204-kvm.qcow2.bz2, md5 627c49a5523fc2cfddebd0a1a396512f), on two separate KVM hosts. Same failure both times, which rules out template or per-host config.

Environment

Component	Version
CloudStack	4.22.0.0 (mgmt + agent)
KVM host OS	RHEL 9.7 (kernel 5.14.0-611.5.1.el9_7.x86_64)
qemu-kvm	9.1.0-29.el9_7.6
libvirt	10.10.0-15.8.el9_7
Zone type	Advanced (isolated tenant networks behind VR)
Primary storage	local
Secondary storage	NFS
CKS Kubernetes version	v1.33.1 (binaries ISO setup-v1.33.1-fresh.iso)
Template tested	cks-ubuntu-2204-kvm.qcow2.bz2 from download.cloudstack.org/testing/custom_templates/ubuntu/22.04/

Symptoms

Inside the guest (control VM, via VNC console)

[1454] cloud-init[874]: Waiting for Binaries directory /mnt/k8sdisk/ to be available, sleeping for 15 seconds, attempt: 97
[1469] cloud-init[874]: Waiting for Binaries directory /mnt/k8sdisk/ to be available, sleeping for 15 seconds, attempt: 98
[1484] cloud-init[874]: Waiting for Binaries directory /mnt/k8sdisk/ to be available, sleeping for 15 seconds, attempt: 99
[1499] cloud-init[874]: Waiting for Binaries directory /mnt/k8sdisk/ to be available, sleeping for 15 seconds, attempt: 100
[1514] cloud-init[874]: Warning: Offline install timed out!
[1514] cloud-init[874]: Failed to enable unit: Unit file kubelet.service does not exist.
...
Cloud-init v. 24.4-0ubuntu1~22.04.1 finished at Mon, 11 May 2026 20:56:13 +0000. Datasource DataSourceCloudStack. Up 1514.89 seconds

So: cloud-init's DataSourceCloudStack works (user-data + SSH keys + hostname all applied), but the binaries ISO contents never show up at /mnt/k8sdisk/ and the setup loop times out.

Management server log

ERROR [c.c.k.c.a.KubernetesClusterStartWorker]
  Failed to setup Kubernetes cluster : test-cluster-32 is not in usable state
  as the system is unable to access control node VMs of the cluster

KVM agent log — patch.sh

We separately see patch.sh timing out every 5 minutes via qemu-guest-agent:

WARN [kvm.resource.LibvirtComputingResource]
  Process [...] for command [.../patch.sh -n i-2-171-VM -c template=domP name=test-cluster-32-control-...
  type=cksnode eth0ip=192.168.100.186 ...] timed out. Output is [].
ERROR [kvm.resource.LibvirtComputingResource] Passing cmdline failed:timeout

And virsh qemu-agent-command i-2-171-VM '{"execute":"guest-ping"}' reports:

error: Guest agent is not responding: QEMU guest agent is not connected

Qemu monitor confirms the host side socket is open but disconnected:

charchannel0: filename=disconnected:unix:/var/lib/libvirt/qemu/i-2-171-VM.org.qemu.guest_agent.0,server=on

The qga issue may be a contributing factor (since it's how patch.sh injects the cloud-init cmdline) — but cloud-init clearly still ran via the metadata-server path, so the user-data delivery itself isn't the blocker. The cluster failure is happening in the ISO-mount step that runs after cloud-init.

What we've tried

1. Custom-patched Ubuntu 22.04 cloud-image (installed qemu-guest-agent, containerd, deps via virt-customize --install; added a manual multi-user.target.wants/qemu-guest-agent.service symlink; experimented with clearing BindsTo= via a systemd drop-in). All variants hit the same failure.
2. Downloaded the canonical CKS template fresh from download.cloudstack.org and registered as a new template. Same failure.
3. Disabled two of the three KVM hosts to pin VMs to the original working host. Same failure.
4. Verified /var/www/html/setup-v1.33.1-fresh.iso exists on secondary storage, registered as ISO template id 227, NFS-mounted at nfs://10.96.32.32/mnt/raid0/secondary.

We've validated that:

• The chardev for qga on the host is open (server=on) and just lacks a guest-side connection.
• No SELinux AVC denials on the host.
• Both KVM machine types used are pc-i440fx-rhel7.6.0 (qemu reports this is deprecated).
• The binaries ISO is present and was attached then detached around the time CKS gave up (per KubernetesClusterStartWorker.Detached Kubernetes binaries from VM).

Questions for the community

1. For CloudStack 4.22 + RHEL 9.7 KVM hosts, is there a known issue with the Kubernetes binaries ISO not appearing at /mnt/k8sdisk/ inside Ubuntu CKS guests? Anyone successfully running CKS on this combo?
2. The setup-loop polls /mnt/k8sdisk/ for 25 min. Where does the canonical CKS template expect that mount to be sourced from (CD-ROM device path inside guest)? Should we see a /dev/sr0 block device, and if not, what would cause it to be missing despite CKS having attached the ISO at the libvirt layer?
3. Is the deprecated pc-i440fx-rhel7.6.0 machine type known to break virtio-serial-based qemu-guest-agent on recent Ubuntu cloud kernels? If so, what's the recommended override (hypervisor.kvm.machine.type global setting)?
4. The qga channel chardev shows disconnected even though both the patched Ubuntu cloud-image AND the canonical CKS template have qga installed. Is there a known wiring issue for the virtio-serial channel name org.qemu.guest_agent.0 on RHEL 9.7 / qemu 9.1?

Happy to provide more logs / mgmt-server traces / virsh dumpxml of the affected VMs on request. Cross-linked from discussion #13056 (original report from @anirudh-kaushal).

Thanks!