Summary
On Cloud Hypervisor, a Linux container that finishes its work does not stop. The guest boots again, so the container never ends.
Why this happens
When the workload exits, urunit (the guest init program) ends the VM with this call:
reboot(LINUX_REBOOT_CMD_RESTART)
This asks the guest to restart, not to power off. QEMU can treat a guest restart as a stop, through the -no-reboot flag. Cloud Hypervisor has no flag like this. On a reset it simply reboots the guest. So the container starts over instead of ending.
Scope
This is the normal path, where the application inside the guest exits on its own. It is not the "stop from outside" path, such as docker stop or a pod restart. So it is separate from the graceful shutdown work in #336. Graceful shutdown handles an outside signal. This issue is about the workload finishing by itself.
Status
I have not reproduced this myself. The Cloud Hypervisor guest does not fully boot on my aarch64 setup, so I cannot run the app-exit path here. I am flagging it as @pmoust's observation, from #336 (comment), and not as something I verified.
Likely fix and its trade-offs
@pmoust notes that changing urunit to:
reboot(LINUX_REBOOT_CMD_POWER_OFF)
fixes it on aarch64. There, PSCI SYSTEM_OFF (the arm64 firmware call that powers the machine off) does not need ACPI.
The change is not safe for every case, though:
- On x86_64, Firecracker does not implement ACPI power management. A power-off would leave the Firecracker process running.
- The reference guest kernel that urunc documents does not enable ACPI either. So QEMU and Cloud Hypervisor guests on x86_64 would not power off reliably.
So the fix also needs guest kernel options per architecture. For example, CONFIG_ACPI_BUTTON on x86, and CONFIG_GPIO_PL061 with CONFIG_KEYBOARD_GPIO on arm64. See @pmoust's #336 (comment) comment for the full list.
Credit: @pmoust for the finding.
Summary
On Cloud Hypervisor, a Linux container that finishes its work does not stop. The guest boots again, so the container never ends.
Why this happens
When the workload exits, urunit (the guest init program) ends the VM with this call:
This asks the guest to restart, not to power off. QEMU can treat a guest restart as a stop, through the
-no-rebootflag. Cloud Hypervisor has no flag like this. On a reset it simply reboots the guest. So the container starts over instead of ending.Scope
This is the normal path, where the application inside the guest exits on its own. It is not the "stop from outside" path, such as
docker stopor a pod restart. So it is separate from the graceful shutdown work in #336. Graceful shutdown handles an outside signal. This issue is about the workload finishing by itself.Status
I have not reproduced this myself. The Cloud Hypervisor guest does not fully boot on my aarch64 setup, so I cannot run the app-exit path here. I am flagging it as @pmoust's observation, from #336 (comment), and not as something I verified.
Likely fix and its trade-offs
@pmoust notes that changing urunit to:
fixes it on aarch64. There, PSCI
SYSTEM_OFF(the arm64 firmware call that powers the machine off) does not need ACPI.The change is not safe for every case, though:
So the fix also needs guest kernel options per architecture. For example,
CONFIG_ACPI_BUTTONon x86, andCONFIG_GPIO_PL061withCONFIG_KEYBOARD_GPIOon arm64. See @pmoust's #336 (comment) comment for the full list.Credit: @pmoust for the finding.