feat(monitors): expose a configurable control socket for each monitor by Anamika1608 · Pull Request #850 · urunc-dev/urunc

Anamika1608 · 2026-07-28T11:46:23Z

Description

Expose each monitor's control socket in the normal (config-file / CLI) boot flow, so the runtime can talk to the VMM after the guest starts (graceful shutdown now, snapshots later). Each monitor still boots exactly as before — this only keeps its control socket open and reachable:

Firecracker: launches with --api-sock while keeping --config-file, so the guest still boots from the config file and the socket stays open (drops only --no-api).
QEMU: adds -qmp unix:<path>,server,nowait alongside the disabled human monitor.
Cloud Hypervisor: adds --api-socket path=<path>.

The socket path is configurable via a socket_path monitor field, shared across all monitors through a common ResolveSocketPath helper, defaulting to a per-container /tmp/<id>.sock. After changeRoot, urunc creates the socket path's directory inside the monitor rootfs so a custom path works; it fails only if the location is invalid (a file already exists on the path).

Related issues

Related to LFx Mentorship (CNCF - urunc: Improve lifecycle management of sandbox monitors) #756

How was this tested?

go build ./..., go test ./pkg/... ./internal/..., gofmt, make lint, and cspell pass; the QEMU BuildExecCmd tests cover the new -qmp argument (Ubuntu 24.04 aarch64 VM, KVM).
Live, through real containerd + nerdctl: Firecracker boots from its config file with the socket reachable (state Running) for both the default and a custom socket_path; QEMU (chttp-qemu-linux-aarch64) boots, the guest serves HTTP 200, launches with -qmp unix:/tmp/qemu.sock,server,nowait, and the QMP socket answers query-status with running.
Cloud Hypervisor is unit-tested only: no aarch64 Cloud Hypervisor image is published in the registry, so its socket could not be exercised live.

LLM usage

claude code (opus 4.8) for the understanding of codebase, approach decisions and reviews

Checklist

I have read the contribution guide.
The linter passes locally (make lint).
The e2e tests of at least one tool pass locally (make test_ctr, make test_nerdctl, make test_docker, make test_crictl).
If LLMs were used: I have read the llm policy.

netlify · 2026-07-28T11:46:29Z

✅ Deploy Preview for urunc ready!

Name	Link
🔨 Latest commit	`fb22e0c`
🔍 Latest deploy log	https://app.netlify.com/projects/urunc/deploys/6a7b8692523ae6000868ac1b
😎 Deploy Preview	https://deploy-preview-850--urunc.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

cmainas

Hello @Anamika1608 ,

thank you for the PR. I have added some comments in the code. Also, some generic comments:

There is still the case where an admin configures the urunc to run the monitors without any socket. This might be for security reasons or because they do not require it. Therefore, if the socket path is not in the configuration, then there should be no socket (no default value).
We need to cleanup the socket path, because in case of a container restart (e.g. pod) the monitor might fail to use the same path.

Anamika1608 · 2026-08-04T08:27:49Z

can you take a look at the new commits i have pushed, done all three changes as you said. @cmainas

cmainas

Hello @Anamika1608 ,

thank you for the updates. I have added some more comments. One thing that we need to have in mind is that the socket path can be anywhere, but it should still be in an accessible places for any user and more importantly do not collide with any other file.

Anamika1608 · 2026-08-10T09:55:30Z

hi @cmainas,

thanks for the review. i made all the changes. here is each one.

1. UsesControlSocket is now a method on the VMM interface.
Before, one function checked the monitor type in a switch. Now the interface asks each monitor directly. Qemu, Firecracker and Cloud Hypervisor answer yes. hvt, spt and hedge answer no. A new monitor cannot build without answering this, so we cannot forget it. I added a small test for all six monitors.

2. The socket directory now uses 0o700 (owner only).
Only the monitor's user needs the directory. It does not need wider access.

3. urunc removes the socket in Delete, not in Exec.
Delete is where urunc cleans up the container, so the socket belongs there too. urunc removes it only if the file is really a socket. So a wrong socket_path that points at a normal file is safe. urunc never deletes that file.

4. Firecracker comment.
I moved it to just before the if/else and made it shorter.

5. Docs.
I removed "for you" and used the third person. I dropped "can be anywhere". I wrote the two rules: the parent folder must be writable by the monitor's user, and the path must not sit over a file that already exists.

your main note (accessible for any user, and no collision).
The socket lives inside each container's own root filesystem. urunc creates it after the pivot into that filesystem. So two containers with the same socket_path do not collide. Each socket sits in its own filesystem, so the same path points to a different real place per container.

Inside one filesystem there is also no overwrite. If a folder in the path is already a file, the make-directory step fails. If a file already sits at the socket path, the monitor fails to bind. In both cases urunc keeps the file that is there.

For "accessible for any user": urunc makes the folder as the monitor's user. A folder that is open to all users, such as /tmp, works for any user. So /tmp is the safe default.

one thing i want to ask.
Delete cleans the socket on a normal restart, which is a delete and then a create. It does not run on a stop and then a start of the same container. For /tmp this is fine, because urunc makes a fresh /tmp on every boot, so no old socket stays. I tested this. A stop and then a start on the same qemu container boots again (HTTP 200) with a new socket. The only open case is a socket_path on a folder that survives the restart. There an old socket could block the bind. If you want that case covered too, i can also remove the socket in Exec before the bind. For now i kept it in Delete only, as you asked.

tested live (aarch64, real containerd and nerdctl):

qemu boots with and without socket_path. The -qmp flag (QEMU's control socket) shows only when set.
firecracker boots with and without socket_path. It uses --api-sock when set and --no-api when not.
a custom socket folder is created as 0o700.
a stop and then a start reuse boots again.
cloud hypervisor is unit-tested only. Its guest does not boot on aarch64 here.
build, gofmt, go vet, commitlint and cspell are all clean.

Expose each monitor's control socket in the normal boot flow, so the runtime can keep talking to the VMM after the guest starts. Every monitor boots exactly as before; the only change is that its control socket stays open and reachable: - Firecracker launches with --api-sock instead of --no-api, keeping --config-file so the guest still boots from the config file. - QEMU exposes a QMP Unix socket in server mode, configured not to wait for a client before booting, alongside the disabled human monitor. - Cloud Hypervisor exposes its REST API socket (--api-socket). The socket location is configurable through a new socket_path option under a monitor's configuration, wired through MonitorConfig, ExecArgs and the state.json annotation passthrough, with a per-container default of /tmp/<id>.sock behind a DefaultSocketDir constant and a shared resolveSocketPath helper. After changeRoot, urunc creates the socket path's directory inside the monitor rootfs, so any custom path works; it fails only if the location is invalid, such as a file already existing at one of the path's components. Extend the QEMU BuildExecCmd tests to cover the new argument and document the socket_path option. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Remove the default /tmp/<id>.sock socket path. A monitor now gets a control socket only when socket_path is set in config; with no socket_path it launches with no control socket, exactly like upstream. Firecracker restores --no-api in that case. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Move the control socket directory creation from between changeRoot and setupUser to right after setupUser, so it runs as the monitor's user and a non-root monitor can create and use it. Also remove a stale socket left at the same path by a previous instance before the monitor binds, so a restart reusing the same socket_path does not fail to bind. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

The QEMU QMP flag string "server,nowait" trips cspell in qemu.go and qemu_test.go. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Address the review on the control-socket configuration: - Make UsesControlSocket a method on the VMM interface instead of a switch over the monitor type in vmm.go. Every monitor now declares whether it exposes a control socket, so a new monitor cannot be added without deciding this (the switch could be silently forgotten). Qemu, Firecracker and Cloud Hypervisor return true; hvt, spt and hedge return false. - Create the control socket directory with 0o700 instead of 0o755. The directory is owned by the monitor's user and only the monitor needs it. - Move the stale socket removal out of Exec into Delete, where the rest of the container teardown happens. Delete removes the socket at its real path inside the monitor rootfs, and only when it is actually a socket, so a misconfigured socket_path over a regular file is never deleted. Removing the unlink from Exec also means the monitor now fails to bind on a pre-existing file instead of urunc deleting it. - Move the Firecracker two-mode comment out of the FIXME block to right before the if/else and trim it. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Drop the claim that socket_path can be anywhere and use third person (remove "for you"). Document that urunc creates the directory after it drops privileges to the monitor's user, so the parent must be writable by that user and the path must not sit over an existing file. Note that urunc removes the socket on delete, so a restart reusing the same path does not find a stale socket. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

cmainas · 2026-08-10T10:48:33Z

Hello @Anamika1608 ,

thank you for the update.

your main note (accessible for any user, and no collision). The socket lives inside each container's own root filesystem. urunc creates it after the pivot into that filesystem. So two containers with the same socket_path do not collide. Each socket sits in its own filesystem, so the same path points to a different real place per container.

Inside one filesystem there is also no overwrite. If a folder in the path is already a file, the make-directory step fails. If a file already sits at the socket path, the monitor fails to bind. In both cases urunc keeps the file that is there.

In the case of block/initrd/no rootfs, urunc makes use of the container's image rootfs, so there is still the case of cllision with a file in the container's image rootfs. For example a file /fummy/file and a socket patch /dummy/file will result to an undeterministic situation (depend on the monitor).

one thing i want to ask. Delete cleans the socket on a normal restart, which is a delete and then a create. It does not run on a stop and then a start of the same container. For /tmp this is fine, because urunc makes a fresh /tmp on every boot, so no old socket stays. I tested this. A stop and then a start on the same qemu container boots again (HTTP 200) with a new socket. The only open case is a socket_path on a folder that survives the restart. There an old socket could block the bind. If you want that case covered too, i can also remove the socket in Exec before the bind. For now i kept it in Delete only, as you asked.

As far as I concern, a docker / nerdctl stop will result to delete. I am not sure if there is any case where a stop does not go over delete. Maybe I am missing something.

Anamika1608 · 2026-08-10T17:56:48Z

hi @cmainas,

thanks. both points are right. i checked each in the VM. here is what i found.

1. collision with a file in the image rootfs.

You are correct, and it is worse. i tested QEMU directly. When the QMP socket path already holds a regular file, QEMU does not fail. It tries to delete the file first, then bind. i saw this when the file was root-owned and QEMU ran as a normal user:

qemu-system-aarch64: -qmp unix:/tmp/collide_reg,server,nowait: Failed to unlink socket /tmp/collide_reg: Operation not permitted

So QEMU calls unlink() on the path before it binds. The file survived in my test only because QEMU could not remove it (wrong owner). Inside the container rootfs, where the monitor can write that path, QEMU would delete the image file and bind its socket there. That is silent data loss. Different monitors may act differently, so we should not depend on the monitor. urunc should enforce the rule itself.

2. does a stop go through delete?

i tested this on containerd + nerdctl. A stop does NOT call urunc delete. i watched urunc's state directory /run/containerd/runc/default/<id> (the one that delete removes):

after run: present.
after nerdctl stop: still present (state.json still there).
after nerdctl start: still present, same container id (reused in place).
after nerdctl rm: gone.

So delete runs on rm, not on stop. A stop and then a start reuse the same state directory, with no delete in between. (A Kubernetes pod restart is a different case: it usually creates a new container, so the state directory is fresh, not this one.)

So delete-only cleanup does not cover a stop/start when socket_path is on a path that survives the restart. For /tmp it is still fine, because /tmp is a fresh tmpfs on every boot.

proposed fix (covers both, and does not depend on the monitor).

Before the monitor starts, when socket_path is set, urunc checks the path:

nothing there: continue.
a socket there (a stale one from a previous run of the same container): urunc removes it, so the monitor binds cleanly. This covers the stop/start case for every monitor, not only the ones that unlink.
a non-socket file there: urunc refuses to start, with a clear error, and does NOT delete it. This protects a real file in the image rootfs.

i keep the delete-time cleanup too, for the normal rm path. This revisits your earlier point about keeping the removal only in delete.

Does this work for you? If yes, i will add the check before launch and update the docs.

cmainas · 2026-08-11T06:37:15Z

Hello @Anamika1608 ,

the container start process should not perform any cleanup. The environment should be clean enough to not create issues. Therefore, if delete operation does not cover all cases (nerdctl), then kill will. However, this will be tricky. In the future, we will change kill to perform a API call to the socket, therefore we need the socket. So, we should remove it after we perform the respective API call.

A docker/nerdctl stop goes through urunc kill with SIGTERM and never runs delete, so the control socket was left behind until the next delete. Remove it on a terminating signal (SIGTERM or SIGKILL) as well, while the monitor is still alive and its socket is reachable at its real path inside the monitor rootfs. Factor the removal into a shared helper used by both the kill path (Signal) and Delete: it resolves the monitor rootfs, and removes the file only when it is an actual socket, so a misconfigured socket_path over a regular file is never deleted. The start path performs no cleanup. The removal in Signal is best-effort and never blocks the kill. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Trim the multi-line comments added for the control socket work down to one or two short sentences. No code change. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Anamika1608 · 2026-08-11T20:37:55Z

hi @cmainas , check the changes now? all good to go?

cmainas

Hello @Anamika1608 , thank you for the changes. I have some more comments:

UsesControlSocket should be renamed to SupportsControlSocket to match the other similar methods.
nit: please avoid using comments like "this function is called form", because they will become stale as soon as the function gets called form another place.

cmainas · 2026-08-12T10:33:02Z

+	// UsesControlSocket reports whether the monitor exposes a control socket
+	// (set through socket_path).


nit: unnecessary comment. Name is self-explanatory.

cmainas · 2026-08-12T10:45:07Z

+		if rmErr := u.removeControlSocket(vmm, vmmType); rmErr != nil {
+			uniklog.Warnf("failed to remove control socket: %v", rmErr)
+		}
+	}


The removal f the socket must take place after the monitor process receives the signal.

cmainas · 2026-08-12T10:45:29Z

+// isLethalSignal reports whether the signal stops the container. Only SIGKILL
+// and SIGTERM count (the signals a stop sends).
+func isLethalSignal(signal unix.Signal) bool {
+	return signal == unix.SIGKILL || signal == unix.SIGTERM
+}


unnecessary function, please inline.

cmainas · 2026-08-12T11:10:40Z

+// removeControlSocket deletes the monitor's control socket, if one is set. It
+// skips a missing path and never deletes a non-socket file. Signal (on a
+// lethal signal) and Delete both use it.
+func (u *Unikontainer) removeControlSocket(vmm types.VMM, vmmType string) error {


We can reduce the arguments in this function by simply moving the first logic in the caller. The caller already has a vmm instance.

cmainas · 2026-08-12T11:27:19Z

@@ -50,6 +50,12 @@ func (ch *CloudHypervisor) UsesKVM() bool {
 }

 // SupportsSharedfs returns true as Cloud Hypervisor supports virtiofs


This comment should be moved over SupportsSharedfs

cmainas · 2026-08-12T11:28:09Z

@@ -792,13 +804,65 @@ func setupUser(user specs.User) error {
 }

 // Signal sends a specified signal to container's init.


This comment should be above Signal function.

cmainas · 2026-08-12T11:34:07Z

+	if socketPath == "" || !vmm.UsesControlSocket() {
+		return nil
+	}
+	sockRealPath := filepath.Join(u.monitorRootfs(), socketPath)


use secureJoin here.

Anamika1608 force-pushed the config-socket-all-monitors branch from 48afaad to 3e02d31 Compare July 28, 2026 12:02

Anamika1608 mentioned this pull request Jul 31, 2026

feat(monitors): graceful shutdown over the control socket. #869

Draft

4 tasks

cmainas added the ok-to-test label Aug 3, 2026

cmainas requested changes Aug 3, 2026

View reviewed changes

Comment thread pkg/unikontainers/unikontainers.go Outdated

Anamika1608 requested a review from cmainas August 4, 2026 08:27

cmainas reviewed Aug 5, 2026

View reviewed changes

Comment thread docs/configuration.md Outdated

Comment thread pkg/unikontainers/hypervisors/firecracker.go Outdated

Comment thread pkg/unikontainers/hypervisors/vmm.go Outdated

Comment thread pkg/unikontainers/unikontainers.go Outdated

Comment thread pkg/unikontainers/unikontainers.go Outdated

Anamika1608 added 8 commits August 10, 2026 15:27

docs(configuration): socket_path is opt-in with no default

a3b66f2

Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

docs(monitors): fix stale socket_path field comments

c19b85c

Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

chore(lint): add nowait to the cspell dictionary

224cf3b

The QEMU QMP flag string "server,nowait" trips cspell in qemu.go and qemu_test.go. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Anamika1608 force-pushed the config-socket-all-monitors branch from 4e0bebb to cd63c77 Compare August 10, 2026 10:00

Anamika1608 force-pushed the config-socket-all-monitors branch from afcb148 to 0ddb627 Compare August 11, 2026 20:27

style(monitors): shorten verbose code comments

fb22e0c

Trim the multi-line comments added for the control socket work down to one or two short sentences. No code change. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>

Anamika1608 requested a review from cmainas August 12, 2026 05:43

cmainas reviewed Aug 12, 2026

View reviewed changes

		// UsesControlSocket reports whether the monitor exposes a control socket
		// (set through socket_path).

		@@ -50,6 +50,12 @@ func (ch *CloudHypervisor) UsesKVM() bool {
		}

		// SupportsSharedfs returns true as Cloud Hypervisor supports virtiofs

		@@ -792,13 +804,65 @@ func setupUser(user specs.User) error {
		}

		// Signal sends a specified signal to container's init.

Uh oh!

Conversation

Anamika1608 commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

How was this tested?

LLM usage

Checklist

Uh oh!

netlify Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for urunc ready!

Uh oh!

cmainas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Anamika1608 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmainas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Anamika1608 commented Aug 10, 2026

Uh oh!

cmainas commented Aug 10, 2026

Uh oh!

Anamika1608 commented Aug 10, 2026

Uh oh!

cmainas commented Aug 11, 2026

Uh oh!

Anamika1608 commented Aug 11, 2026

Uh oh!

cmainas left a comment

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cmainas Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Anamika1608 commented Jul 28, 2026 •

edited

Loading

netlify Bot commented Jul 28, 2026 •

edited

Loading

Anamika1608 commented Aug 4, 2026 •

edited

Loading