Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,77 @@ When using netdevsim inside containers or Kubernetes pods:
- **Kubernetes pods:** The udev rule creates real `/dev/ptpN` device
nodes (not symlinks) so they propagate into containers.

## Local libvirt/KVM VMs (Linux x86_64)

### Prerequisites

1. A Linux x86_64 host with KVM support.
2. Install libvirt, QEMU, and cloud-image-utils:

```bash
sudo apt-get install -y qemu-kvm libvirt-daemon-system virtinst cloud-image-utils
```

3. Add your user to the `libvirt` and `kvm` groups (no root needed after this):

```bash
sudo usermod -aG libvirt,kvm $USER
# Log out and back in for group membership to take effect
```

4. VM disks are stored in `~/.local/share/netdevsim-dkms/`. QEMU needs
search permission (`+x`) on each parent directory. The script grants
this automatically via ACL, but if you hit permission errors, run:

```bash
# Find the QEMU user on your system (libvirt-qemu on Debian/Ubuntu, qemu on RHEL/Fedora)
QEMU_USER=$(id -nu libvirt-qemu 2>/dev/null || id -nu qemu 2>/dev/null)
setfacl -m "u:${QEMU_USER}:x" ~ ~/.local ~/.local/share
```

### Quick Setup (no tests)

`scripts/setup-libvirt-ubuntu.sh` creates a libvirt VM with the DKMS modules
installed and loaded, adds an SSH config entry, and stops — no tests are run.

```bash
./scripts/setup-libvirt-ubuntu.sh

# Then connect:
ssh netdevsim-ubuntu-test
```

### Running ptp-operator Tests

Once the VM is set up, SSH in and run the ptp-operator test suite. Clone the
[ptp-operator](https://github.com/k8snetworkplumbingwg/ptp-operator) repo on
the VM, then run:

```bash
cd ptp-operator/scripts
./run-on-vm.sh --dkms --mode oc,bc,dualnicbc,dualnicbcha,dualfollower <VM_IP> | tee logs.txt
```

This builds and deploys the ptp-operator on a Kind cluster inside the VM and
runs all five test scenarios (ordinary clock, boundary clock, dual-NIC boundary
clock, dual-NIC boundary clock HA, dual follower). Results are streamed to
`logs.txt`.

To run a single scenario:

```bash
./run-on-vm.sh --dkms --mode bc <VM_IP> | tee logs.txt
```

### Managing the VM

```bash
virsh list --all # list VMs
virsh domifaddr netdevsim-ubuntu-test # get VM IP
virsh shutdown netdevsim-ubuntu-test # stop VM
virsh undefine --remove-all-storage netdevsim-ubuntu-test # delete VM
```

## Local UTM VMs (macOS)

### Prerequisites
Expand Down
214 changes: 214 additions & 0 deletions scripts/setup-libvirt-ubuntu.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
#!/bin/bash
#
# Set up a libvirt/KVM VM with the netdevsim DKMS modules installed and
# loaded, without running any tests. Adds an SSH config entry so you
# can connect with just: ssh <vm-name>
#
# Delegates to test-libvirt-ubuntu.sh --skip-tests for the heavy lifting.
#
# Usage:
# ./scripts/setup-libvirt-ubuntu.sh [options]
#
# Options:
# --release 22.04|24.04 Ubuntu release (default: 24.04)
# --ssh-key PATH SSH private key for VM (auto-detected)
# --vm-name NAME libvirt VM name (default: netdevsim-ubuntu-test)
# --ram MiB RAM in MiB (default: 16384)
# --cpus N CPU cores (default: 4)
# --disk-size SIZE qemu-img resize target (default: 40G)
# --network NAME libvirt network (default: default)
# --cleanup Delete VM and remove SSH config entry
# --shell Drop into an interactive SSH shell after setup
#
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SSH_CONFIG="${HOME}/.ssh/config"

# ---------------------------------------------------------------------------
# Parse --vm-name and --cleanup from args (needed before delegation)
# ---------------------------------------------------------------------------
VM_NAME="netdevsim-ubuntu-test"
CLEANUP=false
ARGS=("$@")

i=0
while (( i < ${#ARGS[@]} )); do
case "${ARGS[$i]}" in
--vm-name)
(( i+1 < ${#ARGS[@]} )) && VM_NAME="${ARGS[$((i+1))]}"
;;
--cleanup)
CLEANUP=true
;;
esac
(( i++ )) || true
done

MARKER_BEGIN="# --- netdevsim-dkms managed: ${VM_NAME} ---"
MARKER_END="# --- end netdevsim-dkms: ${VM_NAME} ---"

# ---------------------------------------------------------------------------
# Helper: remove the fenced SSH config block for VM_NAME
# ---------------------------------------------------------------------------
remove_ssh_config_entry() {
[[ -f "$SSH_CONFIG" ]] || return 0
if grep -qF "$MARKER_BEGIN" "$SSH_CONFIG"; then
local tmp
tmp="$(mktemp)"
awk -v begin="$MARKER_BEGIN" -v end="$MARKER_END" '
$0 == begin { skip=1; next }
$0 == end { skip=0; next }
!skip
' "$SSH_CONFIG" > "$tmp"
mv "$tmp" "$SSH_CONFIG"
chmod 600 "$SSH_CONFIG"
echo "==> Removed SSH config entry for '${VM_NAME}'."
fi
}

# ---------------------------------------------------------------------------
# Cleanup-only mode: remove SSH config + delegate --cleanup
# ---------------------------------------------------------------------------
if [[ "$CLEANUP" == true ]]; then
remove_ssh_config_entry
exec "${SCRIPT_DIR}/test-libvirt-ubuntu.sh" ${ARGS[@]+"${ARGS[@]}"}
fi

# ---------------------------------------------------------------------------
# Guard: check for existing SSH config entry
# ---------------------------------------------------------------------------
if [[ -f "$SSH_CONFIG" ]] && grep -qF "$MARKER_BEGIN" "$SSH_CONFIG"; then
echo "WARNING: ${SSH_CONFIG} already has an entry for '${VM_NAME}'."
read -r -p "Remove the old entry and continue? [y/N] " answer
case "$answer" in
[yY]|[yY][eE][sS])
remove_ssh_config_entry
;;
*)
echo "Aborting. Pick a different name:"
echo " $0 --vm-name <new-name> ${ARGS[*]:-}"
exit 1
;;
esac
fi

# ---------------------------------------------------------------------------
# Delegate to test-libvirt-ubuntu.sh --skip-tests, capturing output for IP
# ---------------------------------------------------------------------------
LOGFILE="$(mktemp)"
trap 'rm -f "$LOGFILE"' EXIT

echo "==> Setting up libvirt VM '${VM_NAME}' with DKMS (no tests) ..."

"${SCRIPT_DIR}/test-libvirt-ubuntu.sh" --skip-tests ${ARGS[@]+"${ARGS[@]}"} 2>&1 | tee "$LOGFILE"
TEST_RC=${PIPESTATUS[0]}

if [[ $TEST_RC -ne 0 ]]; then
echo "ERROR: test-libvirt-ubuntu.sh exited with code ${TEST_RC}."
exit "$TEST_RC"
fi

# ---------------------------------------------------------------------------
# Extract VM IP and SSH key from test-libvirt-ubuntu.sh output
# ---------------------------------------------------------------------------
VM_IP="$(grep 'VM IP: ' "$LOGFILE" | awk '{for(i=1;i<=NF;i++) if($(i-1)=="IP:") print $i}' | tail -1 || true)"
SSH_KEY="$(grep 'ssh -i ' "$LOGFILE" | awk '{for(i=1;i<=NF;i++) if($i=="-i") {print $(i+1); exit}}' | tail -1 || true)"

if [[ -z "$VM_IP" ]]; then
echo "WARNING: Could not extract VM IP from output. Skipping SSH config."
exit 0
fi

# ---------------------------------------------------------------------------
# Install test dependencies (but not the test repo itself)
# ---------------------------------------------------------------------------
SSH_OPTS=(-i "$SSH_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=5)
vm_ssh() { ssh "${SSH_OPTS[@]}" "root@${VM_IP}" "$@"; }

echo "==> Installing test dependencies on VM ..."
vm_ssh bash <<'DEPS'
set -euo pipefail

apt-get update -qq
apt-get install -y -qq podman pciutils uidmap slirp4netns openvswitch-switch \
git ethtool linuxptp zsh 2>&1 | tail -5

GOARCH=amd64

# kind
curl -fsSLo /usr/bin/kind "https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-${GOARCH}"
chmod +x /usr/bin/kind

# kubectl + oc
curl -fsSLo /tmp/oc.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz
tar -C /tmp -xzf /tmp/oc.tar.gz oc kubectl
mv /tmp/oc /tmp/kubectl /usr/local/bin/

# Go
GO_VERSION=$(curl -s https://go.dev/VERSION?m=text | head -n 1)
curl -fsSLo /tmp/go.tar.gz "https://go.dev/dl/${GO_VERSION}.linux-${GOARCH}.tar.gz"
rm -rf /usr/local/go
tar -C /usr/local -xzf /tmp/go.tar.gz

# helm
curl -fsSL "https://get.helm.sh/helm-v3.17.3-linux-${GOARCH}.tar.gz" | tar -C /tmp -xzf -
mv "/tmp/linux-${GOARCH}/helm" /usr/local/bin/helm

# PATH for root and ubuntu
for RC in /root/.bashrc /home/ubuntu/.bashrc; do
grep -q /usr/local/go/bin "$RC" 2>/dev/null || \
echo 'export PATH=$PATH:$HOME/go/bin:/usr/local/go/bin' >> "$RC"
done

# inotify limits (kind needs these)
sysctl -w fs.inotify.max_user_instances=512
sysctl -w fs.inotify.max_user_watches=524288

# ginkgo (test runner)
export PATH=/usr/local/go/bin:/root/go/bin:$PATH
go install github.com/onsi/ginkgo/v2/ginkgo@latest

# zsh + oh-my-zsh for root
chsh -s /usr/bin/zsh root
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended
# Carry PATH into .zshrc
grep -q /usr/local/go/bin /root/.zshrc 2>/dev/null || \
echo 'export PATH=$PATH:$HOME/go/bin:/usr/local/go/bin' >> /root/.zshrc

echo "--- Verification ---"
podman --version
kind version
kubectl version --client 2>/dev/null | head -1
go version
helm version --short
ovs-vsctl --version | head -1
ginkgo version
DEPS
echo "==> Test dependencies installed."

# ---------------------------------------------------------------------------
# Add SSH config entry
# ---------------------------------------------------------------------------
mkdir -p "$(dirname "$SSH_CONFIG")"
[[ -f "$SSH_CONFIG" ]] || touch "$SSH_CONFIG"
chmod 600 "$SSH_CONFIG"

# Ensure a trailing newline before our block
[[ -s "$SSH_CONFIG" && "$(tail -c1 "$SSH_CONFIG")" != "" ]] && echo >> "$SSH_CONFIG"

cat >> "$SSH_CONFIG" <<EOF
${MARKER_BEGIN}
Host ${VM_NAME}
HostName ${VM_IP}
User root
IdentityFile ${SSH_KEY}
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
LogLevel ERROR
${MARKER_END}
EOF

echo ""
echo "==> SSH config entry added. Connect with:"
echo " ssh ${VM_NAME}"
Loading
Loading