Restart a stopped pod member by re-mounting its rootfs - #891
Closed
MayCXC wants to merge 25 commits into
Closed
Conversation
The reaper that runs at a concurrency of one deletes everything in the test directory except the unpacked rootfs it means to preserve. It held that rootfs path as FileManager reports it, under /var, and compared it against the entries of contentsOfDirectory, which reports them under /private/var, so the preserved path never matched any entry and the rootfs was deleted along with the per-test files. The unpack coordinator still held it as unpacked, so the next test opened a rootfs that was no longer there and failed with a missing file error. Resolve both sides with resolvingSymlinksInPathWithPrivate, which exists for this difference between the two views of the same directory.
Tests guarded for Linux are compiled out on macOS, so `make test` reports success without having run them, and nothing says so. The target that does run them is not mentioned anywhere outside the makefile.
The default kernel the tests fetch had drifted from the one users run. container installs kata 3.28.0 and its 6.18.15 kernel, while these tests fetched 3.17.0, so the suite exercised a guest with a different feature set to the one it is meant to represent. Nested runtimes are the visible case: 3.17.0 was built without nf_tables, so a docker daemon inside a container fails there and works on what ships. Kata moved from xz to zstd between those releases, so the archive is no longer named for its compression and tar is left to recognise it rather than being told, which also holds if the format changes again.
A test's rootfs clones, init block, and writable layers land in a directory the runner creates for it and deletes when it finishes, so a test holds disk only while it runs at any concurrency. The run-shared directory holds the unpacked images every bootstrap clones from, and is removed when the run ends.
A container's storage is its rootfs, an optional writable layer, and its mounts; a machine's is its containers' plus the volumes they share. One generic shape describes both the Mount values a machine is configured with and the AttachedFilesystem values it reports once attached, so converting between the two is a map over the structure and the roles cannot drift between configuration and attachment. Device addresses are allocated walking the same sorted order the devices are created in, on both backends. Consumers read roles instead of list positions: the spec builders take a container's mounts without prefix arithmetic, pod volumes mount by name, and a cloud-hypervisor volume's virtiofsd is held by a machine-lifetime reference alongside its containers' reference counts.
A container can be given a swap area, so a workload whose memory exceeds its limit is reclaimed rather than killed. The area is a raw file on the host attached as a block device, which is how a guest gets swap it can write to when its own root is read only, and vminitd formats and enables it while mounting the container's filesystems. The area is enabled with discard, and the device backing it is marked non rotational first. virtio block devices are rotational by default, and the kernel only tracks a swap area in clusters when its device is non rotational, so without that the discard flags are accepted and no discard is ever issued. Together they let the sparse file that backs the area release the blocks the guest stops using, rather than holding the area's high water mark for as long as the container runs.
A pod's containers share one virtual machine, so they can share one swap area as well, with the guest kernel deciding whose pages are reclaimed to it rather than each container carrying its own. The area is attached with the pod's other mounts and enabled once, after the agent comes up. A container may cap how much of the area it uses. That cap counts swap alone, while the runtime spec carries memory and swap as a single total, so the container's memory limit is added to it when the spec is built. A cap without a memory limit is rejected when the container is added, because the total cannot be worked out without one. Leaving the cap unset lets a container use the whole area, which is what containers sharing a pool generally want.
A pod works out the memory and swap total its containers are capped at, the way the runtime spec asks for it, and sends that along with the rest of the resources. The guest wrote the memory limit and dropped the rest, so a container drew on the whole of the pod's swap area whatever it had been given. Cgroup v2 wants the swap on its own rather than the total, so the memory limit comes back out of it before the limit is written, and the spec's unlimited and unset values are carried across as the controller spells them. Reading the limit back from the kernel is what the test does, since a limit the guest never applied leaves nothing else to see. https://github.com/opencontainers/cgroups/blob/main/utils.go
An unlimited memory limit leaves nothing to subtract from the memory and swap total, so the swap stands as it was given rather than becoming unlimited alongside it. A memory limit that is negative without being the unlimited sentinel is not a limit at all, and is refused rather than read as one. https://github.com/opencontainers/cgroups/blob/main/utils.go
Copying a file between the host and a container is reachable only for a container in a machine of its own; the same container placed in a pod has no way to be copied into or out of, so a tool that offers `cp` loses it the moment a container joins a pod. What the transfer needs is the machine, the path the container's filesystem sits at in the guest, and a vsock port, and only the path differs between the two cases. It moves to a type that takes those, which both a standalone container and a pod's containers hand it. The round trip is covered against a running pod, including that a file copied into one container reaches that container alone.
A container with a machine of its own is given the size it asked for plus the guest agent's, so a container that asks for a gibibyte has a gibibyte and the agent runs beside it. A pod hands its size to the machine as it stands, so the agent comes out of what the pod asked for and its containers are left with less: a pod given twelve gibibytes reports eleven and a half. The pod's size and the agent's are added the same way, and what a pod was given is what its containers have.
A container with a machine of its own can be given a block device to write into, with its image mounted read-only beneath, so the image is left as it is and every write lands in the layer. The same container placed in a pod has nowhere to be given one. The layer is attached with the container's other block devices, mounted in the guest as the upper layer of an overlay over the image, dropped from the mounts the runtime spec carries, and unmounted with the container. A pod's containers take one whether they were placed before the machine booted or hotplugged into it afterwards.
The file carried the sibling repository's copyright line, so the license header check found no header it recognized.
Stopping a container tears down what it was running and keeps its place: the name still answers for it, and placing another container under it is refused. Removal is the separate act the runtime specification names for giving the place up, taken once the container has stopped, so a name can run again in a machine that outlives what it last ran. https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto
The runtime specification's hooks have a field on the spec type and no way to reach it: the configuration has nowhere to put them and the spec the container is built from never carries any. They travel intact once set, and the bundle's config.json contains them. Whether anything runs them depends on the runtime the container is launched under, which the field's documentation says.
A container in a pod is spawned with the runtime path hardcoded to nil, so the runtime a standalone container reaches through `ociRuntimePath` is out of reach for the same container placed in a pod, and the spec it is built from carries no hooks for that runtime to run. Nothing about a pod prevents either: the runtime path is passed per process, and the pause process is the only one with a reason to stay on the default. The configuration carries both. The container's own runtime path is used when its process starts and when a process is executed in it, and its hooks travel on the spec, as they do for a standalone container.
vmexec builds the container's root without consulting the spec's device list, so a device arrives only if the guest kernel already exposed it in devtmpfs, carrying the kernel's permissions. Those are stricter than what a machine running udev shows: /dev/net/tun and /dev/fuse are 0600 root here and 0666 on any systemd host, which its udev rules set. A container process that is not root therefore cannot open them, which is what a nested rootless runtime needs to do. Create each device the spec names, and set the permissions it asks for on one that is already present, so the spec says what the container sees. LinuxContainer gains the field to populate the list with.
A container is given devices with the permissions it should see them under, and the same container placed in a pod has nowhere to name them, so a device that a machine running udev would show relaxed arrives with the kernel's stricter permissions instead. The configuration carries them and they reach the runtime specification, as they do for a container with a machine of its own.
A machine's storage devices are fixed once it boots, so a container added to a pod that is already running has nowhere for its filesystem to arrive and the attempt is refused outright. The machine's USB controller takes devices while it runs, and mass storage is one of them. The machine is configured with a controller so one exists to attach to, and a provider attaches a disk through it, names it apart from the disks the machine booted with, and detaches it when the container is done. A directory share stays fixed at boot, which is said rather than pretended.
A disk that arrives over the machine's USB controller needs a guest that can see it, and the kernel is built with no USB support at all and no SCSI disk driver, so the disk attaches to a guest with nothing to enumerate it. The controller, mass storage and the SCSI disk it presents as are built in.
The guest already retries a mount that races a device the machine has taken while running, forcing a PCI rescan between attempts, but it did that only for a virtio source. A USB mass storage disk arrives as a SCSI disk on its own bus, which needs no rescan and does need the wait, so it went straight to the failure instead. The rescan is left where it is and the sources it is reached for now include the SCSI disks, so one mechanism covers a device however it arrives.
A hotplug provider holds the machine's mount registry, seeded with what the machine booted with, so that a container added while the machine runs is registered alongside the ones that were there from the start. Cloud hypervisor's machine reads it back through the provider for exactly that reason. Virtualization's machine kept a second registry of its own and answered from that one, so every registration the provider took landed somewhere nothing read. A container added to a running machine was then built with no mounts at all, and its process died reaching for /dev/null after the pivot rather than anywhere near the registry that was missing. The registry belongs to whichever provider is installed, which the protocol now says, so the machine has one to forward to and answers from its own copy only where no provider exists.
The virtiofs device takes a replacement share while the machine runs, so hotplugVirtioFS exports a directory by setting a share carrying it alongside the ones already exported, and releaseVirtioFS withdraws what no container references anymore.
startContainer guarded the created state, so a member that ran and then stopped could only be brought back by rebooting the whole machine, taking every sibling with it. Bringing it back in place is cheap: a pod member's block devices are attached when the machine boots and stay attached while it runs, since they cannot be detached from a running machine, so a stopped member still has its storage; only its guest rootfs mount and its process were torn down. Follow the lifecycle the runtime specification names, the one removeContainer already cites: stopping keeps the member's place, removal gives it up. stopContainer tears down the process and unmounts the rootfs but leaves the block devices attached and the mount registry entry intact; removeContainer detaches the devices and clears the entry. startContainer accepts a stopped member and re-mounts its rootfs before starting a fresh process on it, reusing the block, image, and shares unchanged. Re-mounting leaves the member created, exactly as a freshly placed one, so a failure before the process starts is cleaned up by the same stopContainer path.
Two sleeping members share a pod; one is stopped while the other holds the machine up, then started again and exec'd into to prove it came back on its reused rootfs. The member keeps its block devices across the stop, so the restart is a plain re-mount of the guest rootfs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
startContainerguarded the created state, so a member that ran and then stopped could only be brought back by rebooting the whole machine, taking every sibling with it.Bringing it back in place is cheap: a pod member's block devices are attached when the machine boots and stay attached while it runs, since they cannot be detached from a running machine, so a stopped member still has its storage; only its guest rootfs mount and its process were torn down.
This follows the lifecycle the runtime specification names, the one
removeContaineralready cites: stopping keeps the member's place, removal gives it up.stopContainertears down the process and unmounts the rootfs but leaves the block devices attached and the mount registry entry intact;removeContainerdetaches the devices and clears the entry;startContaineraccepts a stopped member and re-mounts its rootfs before starting a fresh process on it, reusing the block, image, and shares unchanged. Re-mounting leaves the member created, exactly as a freshly placed one, so a failure before the process starts is cleaned up by the samestopContainerpath.Dependency and Merge Order
This targets stock
apple/containerization:main. Merge order: #863, #872, #878, #885, #886, #887, #888, #890, then this. The head carries their commits because all of them target stockmain; after they merge, this branch takes currentmainand its remaining diff is the two commits above.The removal semantics this builds on are in #885.
Motivation and Context
Discussion: #881.
Testing
make containerization,make check, andswift testclean.make integration: two sleeping members share a pod; one is stopped while the other holds the machine up, then started again and exec'd into to prove it came back on its reused rootfs.Type of Change