Bug: DynamicNetwork.NetworkSetup incorrectly blocks container startup when unrelated or stale "tap" interfaces exist in the network namespace

[singhshresth26]

Summary

When spawning a unikernel using the DynamicNetwork configuration, urunc attempts to ensure that only a single unikernel is run inside the current network namespace. It checks for existing TAP interfaces by calling getTapIndex(). If tapIndex > 0, it aborts container startup and throws the following error:
unsupported operation: can't spawn multiple unikernels in the same network namespace

However, the current check in getTapIndex() is too broad because it scans for any network interface containing the substring "tap" in its name:

for _, iface := range ifaces {
    if strings.Contains(iface.Name, "tap") {
        tapCount++
    }
}

Detailed Problem & Scenarios

This simplistic check introduces two main issues:

1. False Positives with Unrelated Interfaces:
   In containerized or Kubernetes environments, CNI plugins or host systems often create virtual interfaces, bridges, or tunnels that contain the word "tap" (such as cni-tap0, vlan-tap, or tap-master). If any such interface is present in the network namespace, getTapIndex() will return 1, causing urunc to falsely believe a unikernel is already running and refuse to spawn the first container.

2. Blocking on Stale Interfaces:
   If a previous urunc container execution crashes or is terminated uncleanly before it can delete its TAP device (tap0_urunc), the dead device remains in the namespace. On subsequent container launch attempts, getTapIndex() will detect the stale tap0_urunc device and fail to start any new containers, instead of reusing the name or cleaning it up.

Affected Code

• getTapIndex() in pkg/network/network.go (around lines 65-80)
• NetworkSetup() in pkg/network/network_dynamic.go (around lines 36-42)

Expected Behavior

urunc should only count TAP interfaces that match the naming pattern of its own created devices (i.e. ^tap\d+_urunc$).

Suggested Fix

Update getTapIndex() in pkg/network/network.go to compile and match the exact regex pattern of urunc tap devices (the same regex used in CleanupAllUruncTaps):

func getTapIndex() (int, error) {
	ifaces, err := net.Interfaces()
	if err != nil {
		return 0, err
	}
	tapRe := regexp.MustCompile(`^tap\d+_urunc$`)
	tapCount := 0
	for _, iface := range ifaces {
		if tapRe.MatchString(iface.Name) {
			tapCount++
		}
	}
	if tapCount > 255 {
		return tapCount, fmt.Errorf("TAP interfaces count higher than 255")
	}
	return tapCount, nil
}

[cmainas]

Hello @singhshresth26 ,

could you provide any scenario which will trigger this issue?

[singhshresth26]

1. A common scenario is when a previously running unikernel/container crashes, or the VMM/host exits abruptly (e.g., force-killed via SIGKILL) before the network cleanup routine (CleanupAllUruncTaps()) can successfully delete the interface.

As a result, the TAP device (like tap0_urunc) is left-over in the namespace. On a subsequent attempt to run the container, getTapIndex() detects this stale interface and blocks container startup with:
unsupported operation: can't spawn multiple unikernels in the same network namespace

2. Running inside containerized or Kubernetes environments where CNI plugins or host routing configurations create virtual network interfaces containing the word "tap" in their name (e.g., cni-tap0, vtap0, or tap-master).

Since getTapIndex() checks for the substring "tap" via strings.Contains(iface.Name, "tap"), these unrelated interfaces increment tapCount. This causes urunc to falsely believe a unikernel is already running, blocking container startup with:
unsupported operation: can't spawn multiple unikernels in the same network namespace

[cmainas]

Hello @singhshresth26 ,

could you also provide any CNI which uses "tap" in the cirtual network interfaces?

[singhshresth26]

1. The official CNI tap plugin: (part of the standard containernetworking/plugins library) which is specifically designed to create a TAP device inside the container namespace.
2. Macvtap CNI: (developed by the k8snetworkplumbingwg) which is widely used in Kubernetes to provide high-performance network interfaces for VM workloads (e.g. KubeVirt).
3. Contiv/VPP CNI: which uses Vector Packet Processing in user-space and configures TAP devices to bridge packet forwarding to the pod namespace.

Even outside of these, any nested container/unikernel orchestration setups or helper VPN/tunnel agents running alongside the workload in the same netns might define tap interfaces.

[singhshresth26]

Hi @cmainas , I would love to contribute the fix. Let me know if I should go ahead and open a PR!

[cmainas]

1. The official CNI tap plugin: (part of the standard containernetworking/plugins library) which is specifically designed to create a TAP device inside the container namespace.

This is is used in order to create tap devices for containers. In the urunc execution model if the container for any reason needs a tap device this needs to be inside the sandbox.

2. **Macvtap CNI:** (developed by the [k8snetworkplumbingwg](https://github.com/k8snetworkplumbingwg/macvtap-cni)) which is widely used in Kubernetes to provide high-performance network interfaces for VM workloads (e.g. KubeVirt).

Broken link.

3. **Contiv/VPP CNI:** which uses Vector Packet Processing in user-space and configures TAP devices to bridge packet forwarding to the pod namespace.

Again, this is for specific workloads which require a tap device.

Even outside of these, any nested container/unikernel orchestration setups or helper VPN/tunnel agents running alongside the workload in the same netns might define tap interfaces.

I am not sure I understand the "nested container/unikernel orchestration".

So far, we have not came across any real-world scenario where this could be a problem. If you use urunc and you cam across such a scenario, please let us know how we can reproduce a setup.

[singhshresth26]

That makes sense. Though even if we don't have a production setup using these CNIs right now, the current check in getTapIndex() (strings.Contains(iface.Name, "tap")) is a bit broad. If a CNI or VPN agent did configure a TAP device (like tap0) as the namespace's primary network interface, urunc would fail to start thinking there's another unikernel running.

We could make this check a lot more robust by matching tap\d+_urunc (similar to how CleanupAllUruncTaps does it). This would prevent false positives from non-urunc TAP interfaces without any risk to the existing flow.

Let me know if you think this is a worthwhile improvement to have, and I can open a PR. If not, I'm happy to close this issue.

[cmainas]

If we ever reach such a use case, we will find a solution. Right now, it is just a hypothesis.

[Nachiket-Roy]

Hello, following up on the discussion above. Point 1 (interfaces) makes sense to leave as hypothetical for now based on the conversation. I did try to reproduce point 2 (stale tap0_urunc after a crash), though
Repro:

[Step 1: Setup & Launch]
Create persistent network namespace (simulating a Kubernetes Pod sandbox)

Launch urunc container with DynamicNetwork
├──> tap0_urunc is created in the netns
└──> Monitor PID is saved to container state (/run/urunc/<id>/state.json)

[Step 2: Confirm Running]
Verify /proc/<PID>/ns/net exists on the host.
Verify tap0_urunc exists in the netns with ip link.

[Step 3: Simulate Ungraceful Crash]

kill -9 <monitor_pid>

Confirm /proc/<PID>/ is gone.

[Step 4: Execute Runtime Cleanup]
Run:

urunc --debug kill <container-id>

The following log is captured:

"could not find sandbox's network namespace: the namespace does not exist"

kill then returns early, so CleanupAllUruncTaps() is bypassed.

Then run:

urunc delete <container-id>

This deletes the state files, but does not perform tap cleanup.

[Step 5: Verify Leaked Resource]
Inspect the netns:

ip -n <netns-name> link show

tap0_urunc is still present and active.

[Step 6: Attempt Container Restart]
Launch a replacement container in the same netns, simulating a container restart or CrashLoopBackOff.

It fails with:

unsupported operation: can't spawn multiple unikernels in the same network namespace

Captured output:

From urunc kill (Step 4):

time="2026-08-13T02:23:54+05:30" level=info msg="could not find sandbox's network namespace: the namespace does not exist" subsystem=unikontainers

From the container restart (Step 6):

failed to setup network: getTapIndex failed: unsupported operation: can't spawn multiple unikernels in the same network namespace

Could you try reproducing this on your end? It seems like a real bug. The tap cleanup is skipped both when kill hits the missing-netns early return and when delete is called directly without a preceding kill, which containerd can do when a task is already STOPPED.

[cmainas]

Hello @Nachiket-Roy ,

thank you for trying to reproduce the issue. I have various questions though:

• Can you also include the commands you used to reproduce the issue?
• In Step 4, urunc fails to enter the network namespace, but you mention that this network namespace is persistent. So if it really is persistent then urunc kill should be able to join it. Keep in mind that urunc create and urunc kill run in the same context. If "urunc create" was successful so should "urunc kill" be. The only difference is if the namespace gets destroyed in the meantime because there is no process inside this namespace and therefore the kernel cleans it up.
• You mention that ip -n <netns-name> link show can enter the network namespace while urunc kill not, how is this possible?
• you have performed a urunc delete, therefore there is no restart. Delete removes state and everything. After than "urunc start" will never work, "urunc create" needs to re-run.

[Nachiket-Roy]

Yes your 4th point makes it clear my repro was flawed (urunc kill does successfully join the namespace and clean up the tap when given a persistent path, exactly as expected) Aside from this, I also checked Delete() specifically, since that's the path containerd actually takes when a task has already crashed/stopped (no kill involved). It looks like Delete() misses the tap cleanup entirely

E2E Test: Network Tap Leak Reproduction

#!/bin/bash
set -e

echo "=== Running 3 Independent Reproduction Trials ==="

# Create mock VMM binary that accepts hypervisor arguments and runs cleanly
cat << 'EOF' > /usr/local/bin/mock-vmm
#!/bin/sh
sleep 1
EOF
chmod +x /usr/local/bin/mock-vmm

for i in 1 2 3; do
  echo "--------------------------------------------------------"
  echo "Trial $i / 3:"
  
  # Setup Pod netns with eth0 and default route (standard CNI pod network setup)
  mkdir -p /var/run/netns
  ip netns add pod-ns-$i
  ip -n pod-ns-$i link add eth0 type dummy
  ip -n pod-ns-$i addr add 10.244.0.5/24 dev eth0
  ip -n pod-ns-$i link set eth0 up
  ip -n pod-ns-$i route add default via 10.244.0.1 dev eth0 onlink

  # Setup urunc config pointing to mock-vmm
  mkdir -p /etc/urunc
  cat <<EOF > /etc/urunc/config.toml
[monitors.qemu]
path = "/usr/local/bin/mock-vmm"
EOF

  # Setup bundle
  BUNDLE_DIR=$(mktemp -d)
  ROOTFS_DIR=$BUNDLE_DIR/rootfs
  mkdir -p $ROOTFS_DIR/bin
  cp /bin/sleep $ROOTFS_DIR/bin/sleep

  cat <<EOF > $BUNDLE_DIR/config.json
{
  "ociVersion": "1.0.2",
  "process": {
    "user": { "uid": 0, "gid": 0 },
    "args": ["/bin/sleep", "1"],
    "cwd": "/"
  },
  "root": { "path": "rootfs" },
  "annotations": {
    "com.urunc.unikernel.unikernelType": "rumprun",
    "com.urunc.unikernel.unikernelVersion": "1.0",
    "com.urunc.unikernel.hypervisor": "qemu",
    "com.urunc.unikernel.binary": "/bin/sleep",
    "io.kubernetes.cri.container-name": "workload-container"
  },
  "linux": {
    "namespaces": [
      { "type": "mount" },
      { "type": "network", "path": "/var/run/netns/pod-ns-$i" }
    ]
  }
}
EOF

  # 1. Start container 1
  urunc --root /run/urunc create -b $BUNDLE_DIR cntr-1-$i >/dev/null 2>&1
  urunc --root /run/urunc start cntr-1-$i >/dev/null 2>&1

  # Verify tap created
  if ip -n pod-ns-$i link show tap0_urunc >/dev/null 2>&1; then
    echo "  [OK] tap0_urunc created during container 1 run"
  else
    echo "  [FAIL] tap0_urunc was not created"
  fi

  # 2. Wait for exit & delete
  sleep 1.5
  urunc --root /run/urunc delete cntr-1-$i >/dev/null 2>&1

  # Verify tap leaked after delete
  if ip -n pod-ns-$i link show tap0_urunc >/dev/null 2>&1; then
    echo "  [OK] tap0_urunc SURVIVED 'urunc delete' (leaked as expected)"
  else
    echo "  [FAIL] tap0_urunc was deleted unexpectedly"
  fi

  # 3. Attempt container 2
  urunc --root /run/urunc create -b $BUNDLE_DIR cntr-2-$i >/dev/null 2>&1
  set +e
  START_ERR=$(urunc --root /run/urunc start cntr-2-$i 2>&1)
  set -e

  if echo "$START_ERR" | grep -q "can't spawn multiple unikernels in the same network namespace"; then
    echo "  [OK] Container 2 start was BLOCKED by leftover tap0_urunc"
    echo "       Log: $(echo "$START_ERR" | grep "can't spawn multiple unikernels")"
  else
    echo "  [FAIL] Container 2 started unexpectedly: $START_ERR"
  fi

  # Clean trial
  ip netns del pod-ns-$i
  rm -rf $BUNDLE_DIR /run/urunc
done

echo "--------------------------------------------------------"
echo ">>> All 3 trials completed with 100% reproduction consistency."

E2E Test Output

=== Running 3 Independent Reproduction Trials ===
--------------------------------------------------------
Trial 1 / 3:
  [OK] tap0_urunc created during container 1 run
  [OK] tap0_urunc SURVIVED 'urunc delete' (leaked as expected)
  [OK] Container 2 start was BLOCKED by leftover tap0_urunc
       Log: time="2026-08-13T11:44:48Z" level=error msg="Failed to setup network :getTapIndex failed: unsupported operation: can't spawn multiple unikernels in the same network namespace. Possibly due to ctr" subsystem=unikontainers
--------------------------------------------------------
Trial 2 / 3:
  [OK] tap0_urunc created during container 1 run
  [OK] tap0_urunc SURVIVED 'urunc delete' (leaked as expected)
  [OK] Container 2 start was BLOCKED by leftover tap0_urunc
       Log: time="2026-08-13T11:44:51Z" level=error msg="Failed to setup network :getTapIndex failed: unsupported operation: can't spawn multiple unikernels in the same network namespace. Possibly due to ctr" subsystem=unikontainers
--------------------------------------------------------
Trial 3 / 3:
  [OK] tap0_urunc created during container 1 run
  [OK] tap0_urunc SURVIVED 'urunc delete' (leaked as expected)
  [OK] Container 2 start was BLOCKED by leftover tap0_urunc
       Log: time="2026-08-13T11:44:53Z" level=error msg="Failed to setup network :getTapIndex failed: unsupported operation: can't spawn multiple unikernels in the same network namespace. Possibly due to ctr" subsystem=unikontainers
--------------------------------------------------------
>>> All 3 trials completed with 100% reproduction consistency.

Unit Test: Kill() vs Delete() Network Cleanup Behavior

package unikontainers

import (
	"bytes"
	"fmt"
	"os"
	"path/filepath"
	"runtime"
	"testing"

	"github.com/opencontainers/runtime-spec/specs-go"
	"github.com/sirupsen/logrus"
	"github.com/stretchr/testify/assert"
	"github.com/urunc-dev/urunc/pkg/unikontainers/types"
	"github.com/vishvananda/netlink"
	"github.com/vishvananda/netns"
	"golang.org/x/sys/unix"
)

// TestKill_WithExplicitNetNsPath_EntersAndCleansUp proves that Kill() DOES clean up the TAP device
// if it is able to join the namespace via an explicit path.
func TestKill_WithExplicitNetNsPath_EntersAndCleansUp(t *testing.T) {
	if os.Geteuid() != 0 {
		t.Skip("Skipping test: requires root / network namespace permissions")
	}

	runtime.LockOSThread()
	defer runtime.UnlockOSThread()

	var logBuf bytes.Buffer
	origOut := logrus.StandardLogger().Out
	origLevel := logrus.StandardLogger().Level
	logrus.SetOutput(&logBuf)
	logrus.SetLevel(logrus.DebugLevel)
	defer func() {
		logrus.SetOutput(origOut)
		logrus.SetLevel(origLevel)
	}()

	originNetns, err := netns.Get()
	assert.NoError(t, err)
	defer originNetns.Close()

	testNetns, err := netns.New()
	assert.NoError(t, err)
	defer testNetns.Close()

	nsDir := t.TempDir()
	nsPath := filepath.Join(nsDir, "netns-test")
	f, err := os.Create(nsPath)
	assert.NoError(t, err)
	f.Close()

	err = unix.Mount(fmt.Sprintf("/proc/%d/fd/%d", os.Getpid(), int(testNetns)), nsPath, "", unix.MS_BIND, "")
	assert.NoError(t, err, "mount netns failed")
	t.Cleanup(func() {
		_ = unix.Unmount(nsPath, unix.MNT_DETACH)
	})

	ethAttrs := netlink.NewLinkAttrs()
	ethAttrs.Name = "eth0"
	ethDummy := &netlink.Dummy{LinkAttrs: ethAttrs}
	err = netlink.LinkAdd(ethDummy)
	assert.NoError(t, err)
	ipn, _ := netlink.ParseAddr("10.0.0.2/24")
	_ = netlink.AddrAdd(ethDummy, ipn)
	_ = netlink.LinkSetUp(ethDummy)

	tapAttrs := netlink.NewLinkAttrs()
	tapAttrs.Name = "tap0_urunc"
	tapLink := &netlink.Tuntap{
		LinkAttrs: tapAttrs,
		Mode:      netlink.TUNTAP_MODE_TAP,
	}
	err = netlink.LinkAdd(tapLink)
	assert.NoError(t, err)

	err = netns.Set(originNetns)
	assert.NoError(t, err)

	deadPid := 999999999
	u := &Unikontainer{
		State: &specs.State{
			ID:  "test-container-explicit",
			Pid: deadPid,
			Annotations: map[string]string{
				annotHypervisor: "qemu",
			},
		},
		UruncCfg: &UruncConfig{
			Monitors: map[string]types.MonitorConfig{
				"qemu": {
					BinaryPath: "/bin/sh",
				},
			},
		},
		Spec: &specs.Spec{
			Linux: &specs.Linux{
				Namespaces: []specs.LinuxNamespace{
					{
						Type: specs.NetworkNamespace,
						Path: nsPath, // Explicit persistent path (as in K8s / CRI)
					},
				},
			},
		},
	}

	err = u.Kill()
	assert.NoError(t, err)

	logOutput := logBuf.String()
	t.Logf("Kill() output with explicit path:\n%s", logOutput)

	assert.Contains(t, logOutput, "Joining network namespace")
	assert.Contains(t, logOutput, "Joined network namespace")
	assert.Contains(t, logOutput, "net cleanup called")

	_ = netns.Set(testNetns)
	_, err = netlink.LinkByName("tap0_urunc")
	assert.Error(t, err, "tap0_urunc should have been deleted by Kill()")
	_ = netns.Set(originNetns)
}

// TestDelete_WithExplicitNetNsPath_DoesNotCleanUpTap proves that Delete() NEVER invokes
// network cleanup, leaving tap0_urunc intact inside the namespace.
func TestDelete_WithExplicitNetNsPath_DoesNotCleanUpTap(t *testing.T) {
	if os.Geteuid() != 0 {
		t.Skip("Skipping test: requires root / network namespace permissions")
	}

	runtime.LockOSThread()
	defer runtime.UnlockOSThread()

	originNetns, err := netns.Get()
	assert.NoError(t, err)
	defer originNetns.Close()

	testNetns, err := netns.New()
	assert.NoError(t, err)
	defer testNetns.Close()

	tapAttrs := netlink.NewLinkAttrs()
	tapAttrs.Name = "tap0_urunc"
	tapLink := &netlink.Tuntap{
		LinkAttrs: tapAttrs,
		Mode:      netlink.TUNTAP_MODE_TAP,
	}
	err = netlink.LinkAdd(tapLink)
	assert.NoError(t, err)

	err = netns.Set(originNetns)
	assert.NoError(t, err)

	tmpDir := t.TempDir()
	u := &Unikontainer{
		BaseDir: tmpDir,
		State: &specs.State{
			ID:     "test-container-delete",
			Status: "stopped",
			Pid:    999999999, // Dead PID
			Annotations: map[string]string{
				annotHypervisor: "qemu",
			},
		},
		Spec: &specs.Spec{
			Root: &specs.Root{
				Path: tmpDir,
			},
			Linux: &specs.Linux{},
		},
		UruncCfg: &UruncConfig{
			Monitors: map[string]types.MonitorConfig{
				"qemu": {
					BinaryPath: "/bin/sh",
				},
			},
		},
	}

	err = u.Delete()
	assert.NoError(t, err)

	// Verify tap0_urunc is STILL PRESENT in the netns after Delete()!
	_ = netns.Set(testNetns)
	link, err := netlink.LinkByName("tap0_urunc")
	assert.NoError(t, err, "tap0_urunc was NOT deleted by Delete() - it survived!")
	assert.NotNil(t, link)
	_ = netns.Set(originNetns)
}

Unit Test Output

=== RUN   TestKill_WithExplicitNetNsPath_EntersAndCleansUp
    tap_lifecycle_test.go:164: Kill() output with explicit path:
        time="2026-08-13T09:57:37Z" level=debug msg="Joining network namespace" path=/tmp/TestKill_WithExplicitNetNsPath_EntersAndCleansUp1289827032/001/netns-test subsystem=unikontainers
        time="2026-08-13T09:57:37Z" level=debug msg="Joined network namespace" subsystem=unikontainers
        time="2026-08-13T09:57:37Z" level=debug msg="net cleanup called" subsystem=network
        time="2026-08-13T09:57:37Z" level=debug msg="cleaning up tap device tap0_urunc" subsystem=network
--- PASS: TestKill_WithExplicitNetNsPath_EntersAndCleansUp (0.01s)
=== RUN   TestDelete_WithExplicitNetNsPath_DoesNotCleanUpTap
time="2026-08-13T09:57:37Z" level=error msg="failed to load monitor resources: open /tmp/TestDelete_WithExplicitNetNsPath_DoesNotCleanUpTap790461403/001/monitor.json: no such file or directory" subsystem=unikontainers
--- PASS: TestDelete_WithExplicitNetNsPath_DoesNotCleanUpTap (0.00s)
PASS

[cmainas]

Hello @Nachiket-Roy ,

thank you for checking once more. It is true that in Delete the network cleanup does not take place. It is a decision to perform the cleanup in Delete too. However, if the issue does not manifest in an actual scenario, I would not give priority to that.