WIP Hypershift On Kubevirt Model 1 localnet-as-primary support #83125
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -136,7 +136,28 @@ fi | |
| oc create namespace "${CLUSTER_NAMESPACE_PREFIX}" --dry-run=client -o yaml | oc apply -f - | ||
| oc create ns "${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}" | ||
| if [[ -n "${ATTACH_DEFAULT_NETWORK}" ]]; then | ||
| if [[ "${ATTACH_DEFAULT_NETWORK}" == "localnet" ]]; then | ||
| LOCALNET_SUBNET="${LOCALNET_SUBNET:-192.168.111.0/24}" | ||
| oc apply -f - <<EOF | ||
| apiVersion: "k8s.cni.cncf.io/v1" | ||
| kind: NetworkAttachmentDefinition | ||
| metadata: | ||
| name: localnet-network | ||
| namespace: ${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME} | ||
| spec: | ||
| config: '{ | ||
| "cniVersion": "0.3.1", | ||
| "name": "physnet", | ||
| "type": "ovn-k8s-cni-overlay", | ||
| "topology": "localnet", | ||
| "netAttachDefName": "${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}/localnet-network", | ||
| "subnets": "${LOCALNET_SUBNET}" | ||
| }' | ||
| EOF | ||
| LOCALNET_ATTACH_DEFAULT="${LOCALNET_ATTACH_DEFAULT:-false}" | ||
| EXTRA_ARGS="${EXTRA_ARGS} --attach-default-network=${LOCALNET_ATTACH_DEFAULT} --additional-network name:${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}/localnet-network" | ||
| else | ||
| oc apply -f - <<EOF | ||
| apiVersion: "k8s.cni.cncf.io/v1" | ||
| kind: NetworkAttachmentDefinition | ||
| metadata: | ||
|
|
@@ -155,10 +176,11 @@ spec: | |
| } | ||
| }' | ||
| EOF | ||
| if [[ "${ATTACH_DEFAULT_NETWORK}" == "true" ]]; then | ||
| EXTRA_ARGS="${EXTRA_ARGS} --attach-default-network=true --additional-network name:local-cluster-${CLUSTER_NAME}/macvlan-bridge-whereabouts" | ||
| else | ||
| EXTRA_ARGS="${EXTRA_ARGS} --attach-default-network=false --additional-network name:local-cluster-${CLUSTER_NAME}/macvlan-bridge-whereabouts" | ||
| fi | ||
| fi | ||
| fi | ||
|
|
||
|
|
@@ -237,4 +259,120 @@ oc wait --timeout=30m --for=condition=Available --namespace=${CLUSTER_NAMESPACE_ | |
| echo "Cluster became available, creating kubeconfig" | ||
| $HCP_CLI create kubeconfig --namespace="${CLUSTER_NAMESPACE_PREFIX}" --name="${CLUSTER_NAME}" >"${SHARED_DIR}/nested_kubeconfig" | ||
|
|
||
| # Post-creation localnet setup: DHCP workaround, port security clearing, ipecho deployment | ||
| if [[ "${ATTACH_DEFAULT_NETWORK}" == "localnet" ]]; then | ||
| LOCALNET_SUBNET="${LOCALNET_SUBNET:-192.168.111.0/24}" | ||
| # Derive gateway IP (.1) from the subnet | ||
| LOCALNET_GW=$(echo "${LOCALNET_SUBNET}" | sed 's|\.[0-9]*/|.1|') | ||
|
|
||
| echo "Waiting for all VMIs to be Running..." | ||
| for i in $(seq 1 60); do | ||
| VMI_RUNNING_COUNT=$(oc get vmi -n "${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}" --no-headers 2>/dev/null | grep -c Running || true) | ||
| if [[ "${VMI_RUNNING_COUNT}" -ge "${HYPERSHIFT_NODE_COUNT}" ]]; then | ||
| echo "All ${VMI_RUNNING_COUNT} VMIs are Running" | ||
| break | ||
| fi | ||
| echo "Waiting for VMIs... (${VMI_RUNNING_COUNT}/${HYPERSHIFT_NODE_COUNT} running) [${i}/60]" | ||
| sleep 10 | ||
| done | ||
|
Comment on lines
+269
to
+277
Contributor
There was a problem hiding this comment. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Fail when all expected VMIs do not start. After the final iteration, the script continues even when fewer than 🤖 Prompt for AI Agents |
||
|
|
||
| echo "Configuring OVN DHCP options and clearing port security for localnet LSPs..." | ||
| for VMI in $(oc get vmi -n "${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do | ||
| NODE=$(oc get vmi "${VMI}" -n "${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}" -o jsonpath='{.status.nodeName}' 2>/dev/null) | ||
| if [[ -z "${NODE}" ]]; then | ||
| echo "WARNING: Could not find node for VMI ${VMI}, skipping" | ||
| continue | ||
| fi | ||
|
|
||
| # Find the OVN pod on the node where the VM is scheduled | ||
| OVN_POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node \ | ||
| --field-selector "spec.nodeName=${NODE}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) | ||
| if [[ -z "${OVN_POD}" ]]; then | ||
| echo "WARNING: No ovnkube-node pod found on node ${NODE} for VMI ${VMI}, skipping" | ||
| continue | ||
| fi | ||
|
|
||
| # Find the localnet Logical Switch Port for this VMI | ||
| LSP_NAME=$(oc exec -n openshift-ovn-kubernetes "${OVN_POD}" -c nbdb -- \ | ||
| ovn-nbctl --columns=name --bare find Logical_Switch_Port \ | ||
| "external_ids:k8s.ovn.org/topology=localnet" 2>/dev/null | head -1) | ||
|
Comment on lines
+295
to
+298
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
file='ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh'
printf '%s\n' '--- target context ---'
sed -n '240,330p' "$file"
printf '%s\n' '--- VMI and LSP-related references ---'
rg -n -C 3 'VMI|Logical_Switch_Port|localnet|DHCP|port.security|LSP_NAME' "$file" ci-operator/step-registry/hypershift/kubevirt 2>/dev/null || true
printf '%s\n' '--- related step files ---'
git ls-files 'ci-operator/step-registry/hypershift/kubevirt/**' | sortRepository: openshift/release Length of output: 50373 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- all references to the target variables and OVN commands ---'
rg -n -C 4 'VMI=|VMI\b|OVN_POD|ovn-nbctl|external_ids:k8s\.ovn\.org|Logical_Switch_Port|set Logical_Switch_Port|clear Logical_Switch_Port' ci-operator/step-registry ci-operator 2>/dev/null | head -n 1200
printf '%s\n' '--- script structure ---'
ast-grep outline ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh || true
printf '%s\n' '--- full script length and relevant assignments ---'
wc -l ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh
rg -n '^[[:space:]]*[A-Za-z_][A-Za-z0-9_]*=' ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.shRepository: openshift/release Length of output: 50374 🏁 Script executed: #!/bin/bash
set -euo pipefail
file='ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh'
printf '%s\n' '--- complete target script in focused chunks ---'
sed -n '1,180p' "$file"
sed -n '181,360p' "$file"
sed -n '361,520p' "$file"
printf '%s\n' '--- workflow and documentation references ---'
rg -n -C 5 'hypershift-kubevirt-create|kubevirt.*create|localnet|network identity|Logical Switch Port' \
ci-operator/step-registry ci-operator/step-registry-config 2>/dev/null | head -n 1600 || trueRepository: openshift/release Length of output: 50374 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- OVN LSP query patterns in repository ---'
rg -n -C 5 'find Logical_Switch_Port|external_ids:k8s\.ovn\.org|Logical_Switch_Port.*external|ovn-nbctl.*find' . \
-g '*.sh' -g '*.yaml' -g '*.yml' -g '*.go' -g '*.py' 2>/dev/null | head -n 2000 || true
printf '%s\n' '--- VMI creation and network configuration commands ---'
rg -n -C 5 'oc create.*vmi|oc apply|virtctl|NetworkAttachmentDefinition|localnet|k8s\.ovn\.org' \
ci-operator/step-registry/hypershift ci-operator/step-registry/kubevirt 2>/dev/null | head -n 2000 || trueRepository: openshift/release Length of output: 50374 🏁 Script executed: #!/bin/bash
set -euo pipefail
file='ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh'
python3 - "$file" <<'PY'
from pathlib import Path
import re, sys
p = Path(sys.argv[1])
lines = p.read_text().splitlines()
for i, line in enumerate(lines, 1):
if 260 <= i <= 330:
print(f"{i:4}: {line}")
print("\nReferences to VMI and LSP in the script:")
for i, line in enumerate(lines, 1):
if re.search(r'\bVMI\b|Logical_Switch_Port|LSP_NAME|localnet', line):
print(f"{i:4}: {line}")
PY
printf '%s\n' '--- file metadata ---'
wc -l "$file"
git diff --stat -- "$file"Repository: openshift/release Length of output: 6770 🌐 Web query:
💡 Result: The file Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for p in Path('.').rglob('hypershift-kubevirt-create-commands.sh'):
print(p)
text = p.read_text()
print(text)
PYRepository: openshift/release Length of output: 15354 🏁 Script executed: #!/bin/bash
set -euo pipefail
url='https://raw.githubusercontent.com/openshift/release/master/ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh'
if command -v curl >/dev/null 2>&1; then
curl -fsSL --max-time 15 "$url" | sed -n '250,330p'
else
echo 'curl unavailable'
fiRepository: openshift/release Length of output: 155 🌐 Web query:
💡 Result: In OVN-Kubernetes, a localnet logical switch port is not a direct attachment for a Virtual Machine Interface (VMI) itself; rather, it serves as the bridge between an OVN logical switch and the physical underlay network [1][2][3]. For a VMI to utilize a localnet topology, the architecture works as follows: 1. Localnet Port Purpose: The localnet port is a specialized logical switch port created on the OVN logical switch, configured with type 'localnet' and an option 'network_name' [4][5][6]. This port connects the logical switch to an Open vSwitch (OVS) bridge on the node, which is mapped to the physical network via 'ovn-bridge-mappings' [2][5][6]. 2. VMI Connectivity: A VMI connects to the logical switch via its own standard VIF (Virtual Interface) logical port, not the localnet port [6]. By attaching to a logical switch that also contains a localnet port, the VMI gains access to the physical network through that logical switch's localnet connection [1][3]. 3. Finding the Port: If you are investigating a specific VMI in an OpenShift environment, you can identify the OVN logical switch port associated with the VMI's interface. Since the VMI's interface port is of standard type (empty string), you will not find the VMI directly on a 'localnet' port [6]. Instead, you should: - Identify the OVN logical switch to which the VMI's port is attached. - List the ports on that logical switch. - Look for the port with 'type: localnet', which provides the connectivity to the physical network for all VIFs on that switch [4][6]. In the context of the openshift/release repository or OVN-Kubernetes controllers, the localnet port is typically managed by the Localnet User Defined Network controller [4]. You can verify the configuration by checking the OVN Northbound database (OVN-NB) for logical switch ports with 'type: localnet' on the specific switch associated with your VMI's secondary network [4][6]. Citations:
🌐 Web query:
💡 Result: In OVN-Kubernetes, the configuration topology=localnet is a specialized network mode used to bridge pod or Virtual Machine (VM) traffic directly to an external physical network [1][2][3]. Topology and Logical Switch Ports When a secondary network is configured with topology: localnet, OVN-Kubernetes creates a cluster-wide logical switch that connects the workloads (pods or VMs) to a physical underlay [4][3]. At the OVN database level, this involves creating a logical switch port (LSP) of type localnet on the logical switch [5]. This specific LSP bridges the OVN logical network to a physical network interface on the node, enabling north-south traffic without requiring SNAT to node IP addresses [1][6]. Key Characteristics 1. Integration: Unlike standard overlay networks that use encapsulation, localnet topology wires the logical switch directly to an existing, pre-configured physical network [7][6]. 2. Configuration Requirements: To function, the OVN-Kubernetes controller requires appropriate mapping of the physical network. This is typically achieved by configuring OVS bridge mappings (e.g., via a NodeNetworkConfigurationPolicy in OpenShift/OKD environments) on the cluster nodes [2][3]. 3. Workload Support: This topology is supported for both Pods and KubeVirt VMs, allowing them to participate directly in the provider's physical subnet [1][3]. 4. VLAN Tagging: If isolation is required on the physical network, a vlanID can be specified in the NetworkAttachmentDefinition (NAD) [2][4]. When this is set, OVN-Kubernetes applies the VLAN tag to the traffic leaving the pod/VM, effectively placing the workload in a specific VLAN on the external network [4]. Logical Structure The Logical_Switch_Port (LSP) for a localnet connection is a distinct OVN construct managed by the OVN-Kubernetes CNI [5]. It is managed by the ovnkube-controller, which monitors Custom Resources (such as ClusterUserDefinedNetwork or NetworkAttachmentDefinition) and translates them into the necessary OVN Northbound database entries [1][8]. These LSPs facilitate the direct exchange of packets between the OVN logical switch and the physical switch port connected to the node [5]. It is important to note that topology: localnet is restricted to secondary networks (it cannot be used for the primary cluster network) and requires consistent configuration of physical network names across the cluster nodes to ensure correct traffic routing [1][9]. Citations:
Select the localnet LSP for the configured network. A localnet LSP is shared by the logical switch, not by an individual VMI. However, this query can select the wrong network when multiple localnet networks exist. Filter by the network identity and require exactly one match. 🤖 Prompt for AI Agents |
||
| if [[ -z "${LSP_NAME}" ]]; then | ||
| echo "WARNING: No localnet LSP found on node ${NODE} for VMI ${VMI}, skipping" | ||
| continue | ||
| fi | ||
|
|
||
| # Create DHCP options with router, DNS server, and lease time | ||
| DHCP_UUID=$(oc exec -n openshift-ovn-kubernetes "${OVN_POD}" -c nbdb -- \ | ||
| ovn-nbctl create DHCP_Options cidr="${LOCALNET_SUBNET}" \ | ||
| options='"lease_time"="3500" "router"="'"${LOCALNET_GW}"'" "server_id"="'"${LOCALNET_GW}"'" "server_mac"="c0:ff:ee:00:00:01" "dns_server"="'"${LOCALNET_GW}"'"' \ | ||
| 2>/dev/null) | ||
|
|
||
| # Bind the DHCP options to the localnet LSP | ||
| oc exec -n openshift-ovn-kubernetes "${OVN_POD}" -c nbdb -- \ | ||
| ovn-nbctl lsp-set-dhcpv4-options "${LSP_NAME}" "${DHCP_UUID}" 2>/dev/null | ||
|
|
||
| # Clear port security so EgressIP-SNATed packets can exit | ||
| oc exec -n openshift-ovn-kubernetes "${OVN_POD}" -c nbdb -- \ | ||
| ovn-nbctl clear Logical_Switch_Port "${LSP_NAME}" port_security 2>/dev/null | ||
|
|
||
| echo "Configured DHCP and cleared port security for VMI ${VMI} on node ${NODE} (LSP: ${LSP_NAME})" | ||
| done | ||
|
|
||
| # Deploy ip-echo on the management cluster with localnet NAD | ||
| IPECHO_NAMESPACE="egressip-ipecho-${CLUSTER_NAME}" | ||
| echo "Deploying ip-echo in dedicated namespace ${IPECHO_NAMESPACE}..." | ||
| oc create namespace "${IPECHO_NAMESPACE}" --dry-run=client -o yaml | oc apply -f - | ||
| oc label ns "${IPECHO_NAMESPACE}" pod-security.kubernetes.io/enforce=privileged --overwrite 2>/dev/null || true | ||
|
|
||
| # Create a localnet NAD in the ip-echo namespace | ||
| oc apply -f - <<IPECHO_NAD_EOF | ||
| apiVersion: "k8s.cni.cncf.io/v1" | ||
| kind: NetworkAttachmentDefinition | ||
| metadata: | ||
| name: localnet-network | ||
| namespace: ${IPECHO_NAMESPACE} | ||
| spec: | ||
| config: '{ | ||
| "cniVersion": "0.3.1", | ||
| "name": "physnet", | ||
| "type": "ovn-k8s-cni-overlay", | ||
| "topology": "localnet", | ||
| "netAttachDefName": "${IPECHO_NAMESPACE}/localnet-network", | ||
| "subnets": "${LOCALNET_SUBNET}" | ||
| }' | ||
| IPECHO_NAD_EOF | ||
|
|
||
| oc apply -f - <<IPECHO_EOF | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: egressip-ipecho | ||
| namespace: ${IPECHO_NAMESPACE} | ||
| annotations: | ||
| k8s.v1.cni.cncf.io/networks: localnet-network | ||
| spec: | ||
| containers: | ||
| - name: ip-echo | ||
| image: quay.io/openshifttest/ip-echo:1.2.0 | ||
| ports: | ||
| - containerPort: 80 | ||
| protocol: TCP | ||
| securityContext: | ||
| runAsUser: 0 | ||
| restartPolicy: Always | ||
| tolerations: | ||
| - operator: Exists | ||
| IPECHO_EOF | ||
|
Comment on lines
+324
to
+365
Contributor
There was a problem hiding this comment. 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift Use the required restricted Pod security controls. Line 325 labels the new namespace as As per coding guidelines, step manifests must not run as root without justification. As per path instructions, Kubernetes manifests require restricted security settings, limits, probes, and a namespace NetworkPolicy. 🤖 Prompt for AI AgentsSources: Coding guidelines, Path instructions |
||
|
|
||
| echo "Waiting for ip-echo pod to be ready..." | ||
| oc wait --for=condition=Ready pod/egressip-ipecho -n "${IPECHO_NAMESPACE}" --timeout=120s | ||
|
|
||
| IPECHO_LOCALNET_IP=$(oc get pod egressip-ipecho -n "${IPECHO_NAMESPACE}" \ | ||
| -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' | \ | ||
| python3 -c "import sys,json; nets=json.loads(sys.stdin.read()); [print(n['ips'][0]) for n in nets if 'localnet' in n.get('name','')]") | ||
| echo "ip-echo localnet IP: ${IPECHO_LOCALNET_IP}:80" | ||
| echo "${IPECHO_LOCALNET_IP}:80" > "${SHARED_DIR}/kubevirt_ipecho_url" | ||
| echo "Localnet post-creation setup complete" | ||
| fi | ||
|
|
||
| echo "${CLUSTER_NAME}" > "${SHARED_DIR}/cluster-name" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Pass a host address to OVN DHCP options.
Line 266 produces
192.168.111.1/24, not192.168.111.1. Lines 307 pass this CIDR value as the DHCP router, server ID, and DNS server. Derive the first host address without the prefix.Proposed fix
📝 Committable suggestion
🧰 Tools
🪛 Shellcheck (0.11.0)
[style] 266-266: See if you can use ${variable//search/replace} instead.
(SC2001)
🤖 Prompt for AI Agents