Connecting OpenShift Virtualization (KubeVirt) VMs directly to a physical L2 network, when Single Node OpenShift runs as a VM on an ESXi host.

Architecture

Physical Network (e.g. 192.168.1.0/24)
    └── ESXi vSwitch (promiscuous mode)
            └── SNO VM (ens224 - second vNIC)
                    └── br-multus (Linux bridge via NMState)
                            └── KubeVirt VM (gets DHCP from physical network)

Step 1: ESXi Configuration

1. Create a new vSwitch in vSphere (e.g. multus)
2. Create a Port Group on that vSwitch (e.g. multus)
3. Add a physical NIC uplink to the vSwitch
4. On the Port Group, set the following security policy:
   • Promiscuous Mode: Accept
   • MAC Address Changes: Accept
   • Forged Transmits: Accept

Promiscuous mode is required so the ESXi vSwitch passes frames destined for KubeVirt VM MACs, which differ from the SNO VM's own MAC.

Step 2: Add a Second vNIC to the SNO VM

1. In vSphere, right-click the SNO VM → Edit Settings
2. Add New Device → Network Adapter
3. Set the network to the multus port group
4. Leave as VMXNET3
5. Click OK (can be done live, no SNO reboot required)

Verify the interface appeared inside SNO via a debug pod:

oc debug node/<node-name>
chroot /host
ip link show

Look for a new interface (e.g. ens224) that is UP with no IP address.

Step 3: Install the NMState Operator

Save the following as nmstate-operator.yaml and apply it:

---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-nmstate
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-nmstate
  namespace: openshift-nmstate
spec:
  targetNamespaces:
    - openshift-nmstate
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kubernetes-nmstate-operator
  namespace: openshift-nmstate
spec:
  channel: stable
  name: kubernetes-nmstate-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

oc apply -f nmstate-operator.yaml
oc get pods -n openshift-nmstate -w

Wait for nmstate-operator-* pod to reach Running.

Then create the NMState instance:

oc apply -f - <<EOF
apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate
EOF

Wait for handler and webhook pods:

oc get pods -n openshift-nmstate -w

All nmstate-handler-*, nmstate-webhook-*, and nmstate-metrics-* pods should be Running.

Step 4: Create the Linux Bridge via NMState

Save the following as multus-bridge.yaml, substituting ens224 if your interface name differs:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: multus-bridge
spec:
  desiredState:
    interfaces:
      - name: br-multus
        type: linux-bridge
        state: up
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: ens224

oc apply -f multus-bridge.yaml
oc get nncp multus-bridge -w

Wait for Available / SuccessfullyConfigured.

Step 5: Create a NetworkAttachmentDefinition (NAD)

Create a NAD in each namespace where VMs need physical L2 access:

oc apply -f - <<EOF
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: multus-physical
  namespace: <your-vm-namespace>
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "multus-physical",
      "type": "cnv-bridge",
      "bridge": "br-multus",
      "macspoofchk": false
    }
EOF

The NAD must be in the same namespace as the VM that uses it.

Step 6: Attach the Network to a KubeVirt VM

In the OpenShift Virt UI, edit the VM and add a secondary NIC pointing to the multus-physical network. The VM will indicate a pending change and reboot to apply it.

Alternatively, add to the VM spec directly:

spec:
  template:
    spec:
      networks:
        - name: physical-lan
          multus:
            networkName: multus-physical
      domain:
        devices:
          interfaces:
            - name: physical-lan
              bridge: {}

Step 7: Configure the Interface Inside the VM (Ubuntu)

After the VM reboots, the new interface will be present but DOWN with no IP:

sudo ip link set enp2s0 up
sudo dhcpcd enp2s0

To make it persistent, create a netplan config:

sudo tee /etc/netplan/99-multus.yaml <<EOF
network:
  version: 2
  ethernets:
    enp2s0:
      dhcp4: true
EOF

sudo netplan apply

The VM will receive a DHCP address directly from the physical network.

Step 7 seems to be only required for ubuntu, when tested with RHEL9/10, Fedora and Windows they all rebooted and found a DHCP IP address automatically.

Notes

• The bridge setup (Steps 3–4) is a one-time per cluster operation
• The NAD (Step 5) is one per namespace
• Adding the NIC (Step 6) is one per VM
• For Kasten blueprints and internal cluster tooling, continue to use cluster DNS (<service>.<namespace>.svc.cluster.local) rather than the L2 IP
• If VMs need stable IPs, configure DHCP reservations on your router using the VM's MAC address