Skillversitas

Networking in a MicroVM

MicroVM uses virtio-net-device for networking. This article covers the various networking modes, bridging, and NAT for connecting VMs to each other.

Network Topology

Internet
    |
[Host]
    |
    +--- [MicroVM 1] (10.0.0.2)
    +--- [MicroVM 2] (10.0.0.3)
    +--- [MicroVM 3] (10.0.0.4)

Mode 1: User Mode Networking

The simplest mode, no root access required. QEMU creates internal NAT:

qemu-system-x86_64 \
  -M microvm -enable-kvm -cpu host -m 256m -smp 2 \
  -kernel vmlinuz \
  -append "console=ttyS0 root=/dev/vda rw ip=dhcp" \
  -drive id=rootfs,file=rootfs.img,format=raw,if=none \
  -device virtio-blk-device,drive=rootfs \
  -netdev user,id=net0,hostfwd=tcp::2222-:22,hostfwd=tcp::8080-:80 \
  -device virtio-net-device,netdev=net0 \
  -nographic -serial stdio

Advantages

  • No root access required
  • Easy port forwarding via hostfwd
  • Automatic DHCP for the guest

Disadvantages

  • The guest cannot be reached from outside the host (except via port forwarding)
  • No communication between VMs
  • Slower performance (user-space networking stack)

Mode 2: TAP Networking

The mode for the best performance and VM-to-VM communication. Requires root access:

# Set up the TAP interface on the host
sudo ip tuntap add dev tap0 mode tap
sudo ip addr add 10.0.0.1/24 dev tap0
sudo ip link set tap0 up

# Run the MicroVM
qemu-system-x86_64 \
  -M microvm -enable-kvm -cpu host -m 256m -smp 2 \
  -kernel vmlinuz \
  -append "console=ttyS0 root=/dev/vda rw" \
  -drive id=rootfs,file=rootfs.img,format=raw,if=none \
  -device virtio-blk-device,drive=rootfs \
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
  -device virtio-net-device,netdev=net0 \
  -nographic -serial stdio

Setting Up the IP on the Guest

# Inside the MicroVM
ip addr add 10.0.0.2/24 dev eth0
ip link set eth0 up
ip route add default via 10.0.0.1

# Test connectivity
ping 10.0.0.1  # Host

Mode 3: Bridge Networking

A bridge connects multiple MicroVMs and the host in a single network segment:

# Set up the bridge on the host
sudo ip link add br0 type bridge
sudo ip addr add 10.0.0.1/24 dev br0
sudo ip link set br0 up

# Create TAP interfaces for each VM
for i in 0 1 2; do
  sudo ip tuntap add dev tap${i} mode tap
  sudo ip link set tap${i} up
  sudo ip link set tap${i} master br0
done

# Run VM 1
qemu-system-x86_64 \
  -M microvm -enable-kvm -cpu host -m 256m -smp 1 \
  -kernel vmlinuz \
  -append "console=ttyS0 root=/dev/vda rw ip=10.0.0.2::10.0.0.1:255.255.255.0::eth0" \
  -drive id=rootfs,file=vm1.img,format=raw,if=none \
  -device virtio-blk-device,drive=rootfs \
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
  -device virtio-net-device,netdev=net0 \
  -nographic -serial stdio

# Run VM 2
qemu-system-x86_64 \
  -M microvm -enable-kvm -cpu host -m 256m -smp 1 \
  -kernel vmlinuz \
  -append "console=ttyS0 root=/dev/vda rw ip=10.0.0.3::10.0.0.1:255.255.255.0::eth0" \
  -drive id=rootfs,file=vm2.img,format=raw,if=none \
  -device virtio-blk-device,drive=rootfs \
  -netdev tap,id=net0,ifname=tap1,script=no,downscript=no \
  -device virtio-net-device,netdev=net0 \
  -nographic -serial stdio

NAT for Internet Access

# Enable IP forwarding on the host
sudo sysctl -w net.ipv4.ip_forward=1

# Set up NAT with iptables
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
sudo iptables -A FORWARD -i br0 -o eth0 -j ACCEPT
sudo iptables -A FORWARD -i eth0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT

Mode 4: macvtap (Bridging Without a Bridge)

Macvtap simplifies bridging without requiring a manual bridge setup:

# Run with macvtap
qemu-system-x86_64 \
  -M microvm -enable-kvm -cpu host -m 256m -smp 2 \
  -kernel vmlinuz \
  -append "console=ttyS0 root=/dev/vda rw ip=dhcp" \
  -drive id=rootfs,file=rootfs.img,format=raw,if=none \
  -device virtio-blk-device,drive=rootfs \
  -netdev tap,id=net0,script=no,downscript=no \
  -device virtio-net-device,netdev=net0 \
  -nographic -serial stdio

Complete Network Setup Script

#!/bin/bash
# Script: setup-microvm-network.sh

set -e

BRIDGE="br0"
SUBNET="10.0.0.0/24"
HOST_IP="10.0.0.1"
NUM_VMS=${1:-3}

# Clean up any previous setup
sudo ip link set $BRIDGE down 2>/dev/null || true
sudo ip link del $BRIDGE 2>/dev/null || true

for i in $(seq 0 $((NUM_VMS - 1))); do
  sudo ip link set tap${i} down 2>/dev/null || true
  sudo ip link del tap${i} 2>/dev/null || true
done

# Create the bridge
sudo ip link add $BRIDGE type bridge
sudo ip addr add ${HOST_IP}/24 dev $BRIDGE
sudo ip link set $BRIDGE up

# Create the TAP interfaces
for i in $(seq 0 $((NUM_VMS - 1))); do
  sudo ip tuntap add dev tap${i} mode tap
  sudo ip link set tap${i} up
  sudo ip link set tap${i} master $BRIDGE
done

# Enable NAT
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -s $SUBNET -o $(ip route | grep default | awk '{print $5}') -j MASQUERADE
sudo iptables -A FORWARD -i $BRIDGE -j ACCEPT
sudo iptables -A FORWARD -i $BRIDGE -m state --state RELATED,ESTABLISHED -j ACCEPT

echo "Network setup complete."
echo "Bridge: $BRIDGE ($HOST_IP/24)"
echo "TAP interfaces: tap0 - tap$((NUM_VMS - 1))"
echo "NAT enabled for internet access"

Troubleshooting Networking

The guest cannot access the internet

# Check IP forwarding
cat /proc/sys/net/ipv4/ip_forward
# Must be 1

# Check the NAT rules
sudo iptables -t nat -L -n -v

DHCP is not working

# In the guest, set up the IP manually
ip addr add 10.0.0.2/24 dev eth0
ip link set eth0 up
ip route add default via 10.0.0.1
echo "nameserver 8.8.8.8" > /etc/resolv.conf

A TAP interface cannot be created

# Check whether the module is loaded
lsmod | grep tun

# Load it if it isn't
sudo modprobe tun