tests: HVT e2e tests fail due to seccomp filter stacking

[namansh70747]

The HVT e2e tests fail due to a seccomp filter stacking issue in pkg/unikontainers/hypervisors/hvt.go. This is reproducible on main and unrelated to any recent changes.

Affected Tests

The following HVT test cases fail across the ctr, nerdctl, and crictl suites:

Suite	Test Name
ctr	Hvt-rumprun-hello
ctr	Hvt-mirage-hello
ctr	Hvt-mirage-block-test
nerdctl	Hvt-rumprun-capture-hello
nerdctl	Hvt-rumprun-ping-redis
nerdctl	Hvt-rumprun-ping-redis-with-block
nerdctl	Hvt-mirage-net
nerdctl	Hvt-rumprun-with-seccomp
nerdctl	Hvt-rumprun-UserGroup
crictl	Hvt-rumprun-redis
crictl	Hvt-mirage-net

Unaffected Backends

All other VMM backends — spt, qemu, firecracker, cloud-hypervisor are unaffected since they do not apply seccomp filters in PreExec().

[cmainas]

Hello @namansh70747 ,

yes, this is a known issue. The problem is that we want to keep the seccomp filter for hvt as tight as possible, but due to Go's execution model, there are cases where some system calls might get executed after the seccomp filter enforcement. This depends on the Go version, distribution, etc..

I am not sure what would be the right approach to resolve this, since we keep adding more and more system calls in the allow list.

[pocopepe]

hey @cmainas, think i found it

FilterFlagTSync in applySeccompFilter() syncs the filter to every Go runtime thread the moment it loads. those threads hit syscalls outside the hvt allowlist and SIGSYS. setupUser() already pins the goroutine with LockOSThread() before PreExec so dropping TSync should be enough, filter still gets inherited by solo5-hvt through execve.

found a second one hiding behind it too: Go's runtime calls prctl(PR_SET_VMA) during heap growth (shows up in the syslog write path), which isn't in the allowlist, so the locked thread itself was crashing between applySeccompFilter and syscall.Exec. adding prctl sorts it.

got both running locally with a regression test for the TSync side. want me to open a pr?

[cmainas]

Hello @pocopepe ,

I am afraid this might be a problem yet again in the future due to some changes in Go's runtime or different distro. And her eis the thing we continue to add more and more system calls in the allow list just to satisfy code, while what we really care about is the hvt binary. Maybe we need to set these seccomp filters from C inside hvt and not in Go. This way we do not need to allow system calls like exec, prctl etc.

[pocopepe]

tried something. basically wrapped prctl + seccomp + execve into one tiny C function with cgo so they all fire together and the Go runtime never sees the filter. think that works? or would a separate C helper be cleaner?

[cmainas]

The we will have problem with dynamic linking against libseccomp. I think we should do that in hvt.

[pocopepe]

looked into the solo5 side, Solo5/solo5#282 has been open since 2020, seccomp for the hvt tender on Linux never landed. Is this what you're looking for? pulling the Go filter without that means hvt runs unrestricted which is worse. are we blocking on that?

[cmainas]

As far as I understand there is a discussion for a seccomp filter in hvt. If this is the case then yes, the hvt knows better which system calls it needs and it can create a good filter without including exec or anything else that is required from any other solution before executing the hvt binary.

[pocopepe]

got a pr up for solo5, something I noticed on running e2e with the compiled binary, Hvt-rumprun-without-seccomp will fail now since the filter always applies for hvt. do we need to look into our tests again if modifying hvt is our way forward?

[cmainas]

If the tests fail, we will update them accordingly.

[pocopepe]

Hey @cmainas, the solo5 PR landed and v0.11.0 is out so gave the approach a go locally. Removed applySeccompFilter() from hvt.go, dropped the go-seccomp-bpf dependency, and removed Hvt-rumprun-without-seccomp as discussed.

Tested with solo5 v0.11.0. Mirage and Spt-rumprun tests all seem to pass. The Hvt-rumprun-* tests crash inside the guest though with Solo5: ABORT: Fatal trap, cr2=0x0 during rumprun init. Swapping in solo5 v0.9.3 brings rumprun up fine, so wondering if the existing rumprun test images were compiled against older solo5 bindings and aren't playing well with v0.11.0? Not sure if that warrants a separate issue or if you'd rather fold it into the CI version bump conversation.

[cmainas]

Hello @pocopepe , these are good news. We can remove the seccomp filters form urunc then. One question though. Does the new support allow to disable seccomp?

The current images target older versions of Solo5 (specifically v0.9.3) as mentioned in the documentation. We will need to update them, but we try to automate this process.

[pocopepe]

Nope, I've hardcoded in the allowlist with what I found on the syscall trace, so there wouldn't be any way to explicitly disable seccomp. On the automation side, are you thinking of a separate repo for the test images similar to monitors-build? Should I raise a pr referencing this issue with regards to the seccomp clean up?

[cmainas]

Hello @pocopepe ,

here is the repository for the automation of the Sample images building (https://github.com/urunc-dev/unikernel-build). It is WIP, but you can check this branch https://github.com/urunc-dev/unikernel-build/tree/ref_build_from_changed_files for the overall structure.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days.