Run every container in a pod, and hold containers, pods, and networks in one core plugin - #2192
Open
MayCXC wants to merge 27 commits into
Open
Run every container in a pod, and hold containers, pods, and networks in one core plugin#2192MayCXC wants to merge 27 commits into
MayCXC wants to merge 27 commits into
Conversation
A container whose workload exceeds its memory limit meets the out of memory killer, with nothing between the two. The runtime it sits on can hand the guest a swap area to reclaim to instead, but nothing here asks for one. Take a size on the command line, keep it on the container's resources beside the memory limit it plays against, and make the area from it at bootstrap: a raw block file in the container's own bundle, alongside the root filesystem, which the guest agent writes a swap header to and enables. The size counts swap alone rather than the memory and swap total the runtime spec carries, so it reads against `--memory` rather than including it. A container that asks for none gets none, which is what one expected to stay within its memory wants. The area holds nothing that outlives the container, so it is made afresh at each bootstrap and the host is told not to synchronize it.
The guest reaches the swap area as a block device, which the kernel takes as a single extent without consulting the host's layout, so the hole-free requirement that binds a swap file inside the guest does not bind the host file backing the device. A sparse backing costs the host only the pages the guest has actually swapped out and gives them back on discard. https://github.com/torvalds/linux/blob/master/mm/swapfile.c
Swap was the only container resource a user could ask for but not configure a default for: --swap existed with nothing behind it, while --cpus and --memory both fall back to their [container] keys. Add [container] swap, defaulting to zero so a container still gets no area unless one is asked for, and resolve it the way the other two resolve.
A container runs in a pod, sharing that pod's machine with whatever else is in it. The pod holds what the machine is: its processors, memory and swap, the networks its containers reach the world through, the hostname they answer to, the resolver and hosts file they read, and the kernel parameters they share. A container holds what is its own: its image, its process, its mounts and its filesystem. That division is the one the container runtime interface draws between a sandbox and the containers in it, and it decides where each setting lives here. A container names the pod it runs in. A caller that names none is asking for a pod of its own and is given a name for it before the container is made, so the rest of the code never has to ask whether a container has one. The runtime reads a bundle to learn what it drives, so a bundle says which of the two it holds, and a pod's carries the machine while the containers placed in it carry their own. https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto
A container ran in a machine of its own, which nothing else could be in. It runs in a pod, and the pod holds that machine. What the machine held for one container a pod holds for the containers in it, so the settings that describe a machine belong to the pod and the operations on a container stay with the container. The pod holds the hostname, the resolver, the hosts file and the kernel parameters. The runtime interface carries all four on the sandbox, and the last of them cannot be a container's: the containers share one kernel, so none of them can set a parameter for itself alone. A container asks for none of them, and a pod given none derives its resolver, its name and its hosts file from the network it attaches to, the way a machine derived them from its own attachments. The pod is written down where the container is, from the same configuration, and the kernel and the init filesystem the machine boots are resolved there already, so the pod is written from those rather than from a second look at the image. A pod named on the command line has to be there before a container names it, since naming one joins it, and joining a pod is joining its network, so the options that describe a network are refused to a container that joins one. The machine comes into being when a container in it starts, which is when a container brought up the machine that was its own. It is asked to run holding the containers the pod holds, one request whether it is coming up around them or is already up and taking in one that is new to it, and the runtime knows which of the two it is from the state it keeps. Stopping a container stops that container. The machine goes down once the last container in it has stopped, which is one rule for any number of them: a machine given a single container goes down when that container stops, and a machine holding several stays up for the rest. Nothing asks how many there are. Stopping one container in a machine is its own call, since the call that stops a machine stops everything in it first. A request names the container it means, so a machine holding one is answered the same way as a machine holding several and no request means "the only one here". https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto
A pod is a thing to make, look at, list, start, stop and remove, so it has the commands the other resources have, and a container is placed in one by name. A pod nobody named is anonymous, the way a volume mounted without a name is, and prune takes those away: a pod someone named is left alone unless all of them are asked for, which is what prune does for volumes.
A volume mounted without a name is made for the container that mounted it and is the container's to keep, so removing the container takes it away. A pod nobody named is the same kind of thing: it holds the machine the container would otherwise have had to itself, and goes with it. A resource someone named is not the container's. It was there before the container and is there after, so a container leaving says nothing about it, and neither does a container leaving a pod that others are still in. https://github.com/containerd/nerdctl/blob/main/pkg/cmd/container/remove.go
A machine is a launchd service, so it outlives the process that registered it. A control plane that starts over a running machine held every pod as not ready and every container as stopped: stops returned through their idempotence guards, deletes removed records while processes ran on, and the next placement under a freed name was refused against the place its stopped predecessor never gave back. The kubelet reconciles the same gap by listing what its runtime holds when it starts, and containerd by re-dialing the shims it finds alive. Before serving, each pod whose service still answers launchd is dialed and believed: the machine's own snapshot says what runs. Containers take their machine's word and are adopted running, tracked by the exit monitor the way bootstrap tracked them first. A stopped container's place is given back to the machine when it is cleaned up, and a pod whose machine still answers refuses a new record under its name. https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto
A volume or network deciding a delete ran its check inside the containers lock through withContainerList, coupling the services at the lock. The decision each caller needs is a named question: the volumes containers mount, the containers holding a volume, the containers attached to a network. Each is answered inside the containers lock and returned, so the answer is as strong as the closure was for reading. What moves out of the lock is the act that followed the answer: a delete now runs after the query, accepting the window a container create can race into, which is the window image delete already accepts; the create that loses names the missing volume or network in its error.
The sandbox domain is one consistency domain: a container bootstraps by asking its pod to run holding it, a pod forced away takes its containers with it, and both claim addresses from the networks they attach. The container-core-containers plugin holds the three services in one process, where those calls keep their lock conventions, the shape CoreImages already gave one plugin holding images and content. The API server keeps what faces the host: health, kernels, volumes, disk usage aggregation, and the DNS server, which resolves container hostnames through the same networkLookup route any client may use. Volumes and disk usage reach containers through named atomic queries over XPC, each decided inside the containers lock and returned, the arrangement every service already used to reach images. Clients of the moved resources dial the plugin's mach service. The routes and their harnesses move unchanged; the plugin boots them the way the API server did, from the same configuration, behind one XPCServer.
An unrecognized subcommand already resolves to a container-<name> plugin binary and execs it, which is how k8s ships. pod and run are the same kind of surface: each plugin re-exposes commands the ContainerCommands library already makes public, so the binaries are declarations with no logic of their own. The pod command's single-letter alias does not survive the move: the compiled-in command list carried it, and plugin resolution goes by the one name the binary is installed under.
A pod's machine boot starts the containers inside it, and each with --ssh forwards the agent named by the donation the boot delivers. The start paths a single container takes collect SSH_AUTH_SOCK into the boot's dynamic environment, and pod start now does the same: the CLI reads the caller's socket, the client sends it with the start message, and the harness hands it to the service, whose machine bootstrap already places every bundle with the boot's environment. A pod booted by pod start behaves like one booted through a member's own start.
A machine nobody named exists because its one container needed it, and it held the container's devices, its named volumes among them, from boot. A machine held past its container's exit keeps those claims: the virtual machine's helper process stays alive with the volume images open, so the next container attaching the same named volume is refused with an invalid storage device attachment. The boot request now says which kind of machine it is booting: one made for its container stops with the last thing in it, releasing what it held, while a machine someone named outlives its members the way a pod's machine does. https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto
The exit handler runs its body in a sendable closure, which reads the actor's properties the way any outside caller would.
A container created with no DNS flags carries a configuration naming no resolver, and the network fills it out: the gateway resolves for the machine's containers. A container created with --no-dns carries no configuration at all, and the machine boots without one, so the guest gets no resolv.conf; filling that case from the network turned the explicit refusal into the default it refused.
The machine is believed only while it answers running, and an answer of anything else takes the service down so a fresh machine can boot. A query that fails carries no such answer: treating it as the machine gone deregistered a live client and booted a second machine against devices the first still held, which failed at its attachments and left the pod unmanageable. The start now fails on the unanswered query and the held client stands, the way kubelet recreates a pod sandbox only on a positive verdict (absent, duplicated, or a status of not ready) and returns a failed status query as the operation's error. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/util/util.go
The network's stored configuration was replaced at every start with a computed one naming no address range, so the range was assigned afresh each time and every guest that outlived the restart held an address, a route, and a resolver belonging to a range that no longer existed. The range now rides through the refresh, and a network that named none is written down with the one it was given, so it asks for the same range from then on.
Nothing recorded is a refusal only where a refusal can be made: a container declines DNS with --no-dns and the machine made for it carries that refusal, while a pod is never asked and so records nothing when it is created without DNS flags. Reading that silence as a refusal left every guest in a named pod without a resolver, holding whatever its root filesystem was last written with, so a machine that came back on a new address range kept answering to the old one. A named pod is given the network's resolver, a pod that names one keeps it, and a container's own machine still declines when the container did. A pod created now records what it was told either way.
A vmnet network's address range is reserved for as long as the network object lives, so a helper that goes away still holding it leaves the range spoken for by a network nobody holds: no interface, no route, no process to point at, and every later attempt on that range refused. The helper releases the network when it is asked to stop, which is where a range is given back for the asking. A range that cannot be taken is then survivable rather than fatal. The range the default network asks for is the one it was given last time, a preference and not a demand, so a default network that cannot take it comes up on whatever is free and says which range it lost; a network someone asked for by name still fails, since that range is the request. Calls to a network helper wait a bounded time, so a helper that never answers is reported instead of leaving every command that needs a network waiting forever with nothing to show.
The network arrives as a plain pointer rather than a managed object, so the retain that comes with it is this side's to balance when the reservation is given up.
Running a foreign architecture and exposing nested virtualization are the machine's to do, and the machine is the pod's, booted with or without them before a container joins. Asking for either as a container joining a pod was accepted and ignored, so a container that needed one started in a machine that could not give it. Both are refused the way the network options already are, and an image of a foreign architecture joining a pod whose machine cannot translate says so instead of failing where the reason is no longer visible. https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto Calls to a network helper take the timeout the calls to the api server take, so a caller that wants to wait as long as it takes still can.
Containers in a pod share one kernel, so a kernel parameter is the machine's and the runtime interface carries it on the sandbox alone. A container carried a field for them that nothing wrote and nothing read, copied into the sandbox made for a container that named no pod, which described a setting a container could never have. https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto
An address is what a host is known by on its network: resolver entries, hosts files, and the caches on the other side of it all name it, and an IPv6 address is derived from the hardware address, so a host that comes back with new ones comes back as somebody else. Both were minted afresh every time, the addresses out of a rotating allocator whose record of who had what lived only as long as the process, so a restart shuffled them among the hosts that came back. What a host was given is written down beside the network that gave it, and a host attaching again is given it back when it is still free, falling to the next free address when it is not. A host-local allocator keeps its allocations the same way, under a directory of its own. https://cni.dev/plugins/current/ipam/host-local/
A network hands out addresses; where what it handed out is written down is the plugin's to say, since it knows where its own state belongs. The allocator asks a store for what was given before and tells it what it gives, and the vmnet plugin keeps those under the network's own directory in the entity store the rest of the plugin state uses, one lease to an entry, the way a host-local allocator keeps its allocations. https://cni.dev/plugins/current/ipam/host-local/
Leaving the wait is leaving the network, whether the wait ended by being asked to stop, by an error, or by returning. Releasing on the asking alone left the other ways out holding the range, so a helper that failed or finished left it held by nobody and the next network asking for that range was refused, with no interface, route, or process to point at.
A store reads what it holds as it opens, and reading a directory that is not there fails, so every caller had to make one first and a caller that forgot took down whatever depended on it: the network helper came up, announced its network, and died on a missing directory, leaving every call that needed a network waiting on a helper that kept restarting. The directory belongs to the store, which makes it as it opens, and the callers that were making it no longer do.
A pod clones the init image's filesystem when it is made and boots that clone for as long as it lives, so the image it came from is the only account of which agent its containers talk to. A container records the image it was made from and a caller compares that against the store to decide the container is stale; a pod held nothing to compare, so a machine could outlive any number of guest rebuilds while still booting the generation it was made from.
This was referenced Aug 27, 2026
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type of Change
Motivation and Context
Closes #2191, which carries the design discussion and the full list of what this fixes. This is large (24 commits) and I would rather agree a split than have it reviewed as one lump; the issue asks that question and I will restructure to whatever answer it gets. The bug fixes listed there are separable and several are independent of pods entirely.
A container runs in a pod. A container ran in a machine of its own, which nothing else could be in. It runs in a pod, and the pod holds that machine. What the machine held for one container a pod holds for the containers in it, so the settings that describe a machine belong to the pod and the operations on a container stay with the container: the pod holds the hostname, the resolver, the hosts file and the kernel parameters. The runtime interface carries all four on the sandbox, and the last of them cannot be a container's, since the containers share one kernel and none of them can set a parameter for itself alone. A pod given none of them derives its resolver, its name and its hosts file from the network it attaches to, the way a machine derived them from its own attachments.
A pod is a thing to make, look at, list, start, stop and remove, so it has the commands the other resources have, and a container is placed in one by name. A pod nobody named is anonymous the way a volume mounted without a name is, and
prunetakes those away while leaving a pod someone named unless all of them are asked for, which is what prune does for volumes. A volume mounted without a name is the container's to keep, so removing the container takes it away, and a pod nobody named goes the same way; a resource someone named was there before the container and is there after. https://github.com/containerd/nerdctl/blob/main/pkg/cmd/container/remove.goOne consistency domain, one plugin. The sandbox domain is one: a container bootstraps by asking its pod to run holding it, a pod forced away takes its containers with it, and both claim addresses from the networks they attach. The
container-core-containersplugin holds the three services in one process, where those calls keep their lock conventions, the shapeCoreImagesalready gave one plugin holding images and content. The API server keeps what faces the host: health, kernels, volumes, disk usage aggregation, and the DNS server, which resolves container hostnames through the samenetworkLookuproute any client may use. Volumes and disk usage reach containers through named atomic queries over XPC, each decided inside the containers lock and returned, the arrangement every service already used to reach images, replacing awithContainerListclosure that coupled the services at the lock.podandrunbecome CLI plugins the way an unrecognized subcommand already resolves to acontainer-<name>binary, each re-exposing commandsContainerCommandsalready makes public.The lifecycle bugs it fixes. Each is reproducible on
mainand several are independent of pods:host-localkeeps its allocations: https://cni.dev/plugins/current/ipam/host-local/--no-dnswas undone for a container in a named pod, because a pod created without DNS flags recorded nothing and that silence was read as a refusal.Marked breaking because
podandrunmove to plugin binaries (thepodcommand's single-letter alias does not survive the move, since plugin resolution goes by the one name the binary is installed under) and because the moved services are reached at the plugin's mach service rather than the API server's.Dependency and Merge Order
The guest half is in Containerization: apple/containerization#885 for the pod capability gaps, alongside #877, #880, #881. This branch also carries #2182's three swap commits, since both target stock
main.Testing
Integration suite: 397 passed. Unit suite: 772 passed.
make fmt,make checkclean. The control-plane adoption path was exercised by restarting the plane over running pods and confirming their containers are adopted running rather than recorded stopped.