I use FreeBSD for work because my clients deploy servers on it. At home, I have a PC with 32 GB of RAM and use NixOS, so I wanted to run FreeBSD locally for quick tests.

My first choice would normally be VirtualBox, but on NixOS it’s a pain: every system upgrade forces VirtualBox to be recompiled. Since I upgrade often, that became unmanageable.

People in the fediverse suggested libvirtd, so I gave it a try. It’s trickier at first, but once you learn a few commands it’s not bad at all—and in fact, it allows for a lot of automation.

Installing Libvirtd

In configuration.nix you need to make the following changes.

# Add the user to these groups, internet wisdom is not clear about what exactly is the name of each group. 

users.users.maikel = {
  extraGroups = [ "libvirtd", "libvirt", "qemu-libvirtd" ];
};

# Enable libvirtd
virtualisation = {
  libvirtd = {
      enable = true;
      qemu.vhostUserPackages = with pkgs; [ virtiofsd ];
    };
};

# Add Virt-Manager makes simpler to explore actual configs. 
programs.virt-manager.enable = true;


environment = {
  systemPackages = with pkgs; [
    virt-viewer
    cloud-utils
    cloud-init
    ];
};

Running commands when using Libvirtd as a system's service

Despite all the changes there, I still need to prepend all virsh and virt-viewer commands with

# For Virsh
virsh -c qemu:///system

# For Virt-viewer
virt-viewer -c qemu:///system

Either that or put sudo ahead of all virsh commands. Since I'm lazy I prefer to do what works every time and since I use Fish shell, I just added to ~/.config/fish/conf.d/abbreviations.fish

abbr --add virt-viewer "virt-viewer -c qemu:///system"
abbr --add virsh "virsh -c qemu:///system"

That way every time I write either it auto completes it. If you don't want to autocomplete either use sudo (for virsh) or the whole -c qemu:///system for virt-viewer (can't use sudo, you need the display). I like abbr instead of alias because with abbr I don't actually forget the full command ever. It's shown to me every time.

Configuring the Network

By default there's nothing running network wise. So you need to start the default network with

# Starts it if you can't see any new networks in ifconfig
virsh -c qemu:///system net-start default

# So it autostarts
virsh -c qemu:///system net-autostart default

That will run the virtual network every time the PC reboots.

Modifying the network to assign fixed IPs

virsh net-dumpxml default > mynetwork.xml

We get this from it on the file mynetwork.xml

<network>
  <name>default</name>
  <uuid>07c8b831-3fa7-4bb1-ae07-fad64b672a67</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:9f:f9:f6'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

I'm going to change the range to 192.168.100.0/24 and add a fixed IP by MAC. Also changed the network name to maikenet

<network>
  <name>maikenet</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:9f:f9:f6'/>
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
      <host mac="52:54:00:6b:3c:58" name="freebsd1" ip="192.168.100.100"/>
    </dhcp>
  </ip>
</network>

Now let's destroy the network interface and create a new one, we're going to need the MAC and name later on.

# To destroy it
virsh net-destroy default

# We need to undefine it in case something is assigned to it already
virsh net-undefine default

# To recreate it from file
virsh net-define mynetwork.xml

# Now start it and autostart to ensure it starts with NixOS
virsh net-start maikenet
virsh net-autostart maikenet

# Check with an ifconfig
ifconfig
# You should see an adapter virbr0 with the right IP

Creating a cloud-init enabled image

For a ZFS VM image

This will create a template on your PC to run cloud-init from the ZFS VM image that uses as ZFS filesystem, the most common case.

# Make a folder for your vms
mkdir $HOME/vms
cd $HOME/vms

# Download standard VM image and unzip it
wget https://download.freebsd.org/releases/VM-IMAGES/14.3-RELEASE/amd64/Latest/FreeBSD-14.3-RELEASE-amd64-zfs.qcow2.xz

# Decompress but keeps the original
xz -dk FreeBSD-14.3-RELEASE-amd64-zfs.qcow2.xz

# Make the disk slightly bigger
mv FreeBSD-14.3-RELEASE-amd64-zfs.qcow2 freebsd14-cloud-init-zfs.qcow2
qemu-img resize freebsd14-cloud-init-zfs.qcow2 10G

# Run it with the "default" network to install CloudInit
virt-install \
  --connect qemu:///system \
  --name freebsd-zfs \
  --memory 2048 \
  --vcpus 2 \
  --disk path=freebsd14-cloud-init-zfs.qcow2,format=qcow2,bus=virtio \
  --os-variant freebsd14.0 \
  --import \
  --network network=maikenet,model=virtio \
  --graphics spice

Now inside the machine

The default root user is passwordless so if you use root it won't ask for any password, just log you in.

# OPTIONAL: Keyboard to Spanish, symbols are in different places
kbdcontrol -l es

# Now inside the machine prepare it for cloud-init (as root, no pass)
pkg update
pkg search cloud-init
pkg install -y WHATEVER_VERSION_YOU_GOT_FROM_SEARCH

# Now enable it
sysrc cloudinit_enable="YES"
poweroff 

# On your host system: Back it up
xz -k freebsd14-cloud-init-zfs.qcow2

Using your own templates to launch custom-made VMs easily

Create a cloud-init config

1. Create this file as user-data.yaml on the $HOME/vms folder

#cloud-config
hostname: freebsd1 # Needs to match the one assigned in the network
users:
  - name: maikel
    shell: /usr/local/bin/fish
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
  # Use mkpasswd -m sha-512 to get this
    passwd: "$6$L80IKTwDwcf......sH0"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz.....maikel.dev
ssh_pwauth: True
keyboard:
  layout: es
packages:
  - fish
  - sudo
  - mkpasswd
  - neovim
  - ncdu
  - git
runcmd:
  # Enable SSH
  - sysrc sshd_enable=YES
  - service sshd start

  # OPTIONAL: Set Spanish keyboard permanently
  - sysrc keymap="es.kbd"
  - service syscons restart

  # OPTIONAL: set root and maikel shells to fish explicitly
  - pw usermod root -s /usr/local/bin/fish
  - pw usermod maikel -s /usr/local/bin/fish
  
  # OPTIONAL: Auto resize main partition
  - gpart recover vtbd0
  - gpart resize -i 4 vtbd0 
  - zpool online -e zroot /dev/vtbd0p4 

2. Use this command to create a CD-ROM ISO to launch it from. Assuming you're in $HOME/vms

cloud-localds seed.iso user-data.yaml

Creating the final machine using the ZFS image

The instructions are prety much the same for a UFS image you just change the filenames.

# Access the folder of your vms
cd $HOME/vms

# Decompress the fresh cloud-init-enabled version we created before. 
xz -dk freebsd14-cloud-init-zfs.qcow2.xz

# Rename it to something more useful to distinguish it from the template
cp freebsd14-cloud-init-zfs.qcow2 freebsd1.qcow2

# Create the seed ISO from user-data.yaml in case you've made any changes.
cloud-localds freebsd1.iso user-data.yaml

# Make the disk bigger here it is set to 20G but you can do whatever size you like
qemu-img resize freebsd1.qcow2 50G

# Install notice here we use the MAC assigned in the network and the hostname also assigned in the network, they both have to match.
virt-install \
  --connect qemu:///system \
  --name freebsd1 \
  --memory 4096 \
  --vcpus 4 \
  --disk path=freebsd1.qcow2,format=qcow2,bus=virtio \
  --disk path=freebsd1.iso,device=cdrom \
  --os-variant freebsd14.0 \
  --import \
  --network network=maikenet,model=virtio,mac=52:54:00:6b:3c:58 \
  --graphics spice \
  --noautoconsole # Added this here because I prefer to let it run first. 
 
# To view
virt-viewer freebsd-new

And that's it, your system should be up and running ready to be used and because you assigned it the name and MAC expected by your new network, it should stick to the IP you see it with. 🥳

Extra steps for your own sanity

Resizing the partition to use all available space (ZFS)

If you want your system to use all the available space in your qcow2 file after resizing it you'll need some extra steps. This can all be added on the user-data template though which I did so you don't need to.

# Ensure vtbd0 is the name of it
gpart show

# Reize partition
gpart recover vtbd0

# Get the lice or number of partition, in my case is 4
gpart show 

# This is assumign the slice is 4
gpart resize -i 4 vtbd0 

# Again the end "p4" depends on the slice number
zpool online -e zroot /dev/vtbd0p4 

That's all your machine is ready to use. If you ever need to change the size of the qcow2 file repeat those steps.

Autostart this machine with NixOS with Nixos

Run on the host machine

virsh --connect qemu:///system autostart freebsd1

Detach cloud-init disk just in case

Normally cloud-init runs only once, but just to be sure on the host machine

# To find the name of the ISO device, in my case "hda"
virsh --connect qemu:///system virsh domblklist freebsd1

# To both remove it and ensure it never comes back after reboot
virsh --connect qemu:///system change-media freebsd1 hda --eject --config --live 

SSH-ing in made easier with Fish shell functions

I don't want to have to finding the IP before I connect so I wrote this fish shell function stored in ~/.config/fish/functions/sshvm which uses the global variable $LOCAL_VM_KEY as path to the private key used to log into the FreeBSD VM. I defined that key in ~/.config/fish/conf.d/variables

function sshvm
	# This gets the IP of the server with the given name
    set ip (virsh --connect qemu:///system domifaddr $argv[1] | string match -r '\d+\.\d+\.\d+\.\d+' | head -n1)
    # This connects to the server using the private key
    ssh -i $LOCAL_VM_KEY maikel@$ip
end

Then I just use

sshvm freebsd-new

Or whatever name I gave to that virtual machine.

Add VMs quickly to the network to get fixed IP

With this fish function you recreate the network to assign a fixed IP to any new machine by passing it the hostname you'll be using on it, you should save the resulting MAC address.

function add_vm_to_network
    if test (count $argv) -lt 1
        echo "Usage: add_vm_to_network <hostname>"
        return 1
    end

    set vmname $argv[1]
    set net maikenet

    # Dump network XML
    set netxml (mktemp)
    virsh -c qemu:///system net-dumpxml $net > $netxml

    # Find the highest existing IP
    set last_ip (grep -oP "ip='\K[0-9.]+" $netxml | sort -t. -k4,4n | tail -n1)

    if test -z "$last_ip"
        echo "No static host entries found, starting at 192.168.100.100"
        set new_ip 192.168.100.100
    else
        set base (echo $last_ip | cut -d. -f1-3)
        set last_octet (echo $last_ip | cut -d. -f4)
        set new_octet (math $last_octet + 1)
        set new_ip "$base.$new_octet"
    end

    # Generate a random MAC (QEMU OUI prefix)
    set mac 52:54:00:(openssl rand -hex 3 | sed 's/\(..\)/\1:/g; s/:$//')

    # Insert new host entry before </network>
    sed -i "/<\/dhcp>/i \ \ \ \ \ \ <host mac='$mac' name='$vmname' ip='$new_ip'/>" $netxml

    # Apply the new network XML
    virsh -c qemu:///system net-define $netxml
    virsh -c qemu:///system net-destroy $net
    virsh -c qemu:///system net-start $net
    virsh -c qemu:///system net-autostart $net

    echo "✅ Added host:"
    echo "  Hostname: $hostname"
    echo "  MAC: $mac"
    echo "  IP: $new_ip"

    rm $netxml
end

# Usage: add_vm_to_network freebsd3

Creating machines quickly with Fish function

This requires first a few manual steps but simplifies creation

1. Either use the previous fish function to get the MAC or manually edit the network, destroy and relaunch with virsh net-edit maikenet to add a MAC and the hostname.
2. edit the $vm-user-data.yaml file for that machine, setting the hostname. For example if the machine is going to be called freebsd3 you should first run

# Previous steps

# Add VM to the network and annotate the MAC
add_vm_to_network freebsd3

# Copy user-data.yaml into VM-user-data.yaml
cp $HOME/vms/user-data.yaml $HOME/vms/freebsd3-user-data.yaml

# Edit the resulting file to have in hostname the machine hostname you desire
vi $HOME/vms/freebsd3-user-data.yaml

Then create the machine

function createvm
    if test (count $argv) -lt 2
        echo "Usage: createvm <vm-name> <mac-address>"
        return 1
    end

    set vm $argv[1]
    set mac $argv[2]

    set vm_dir $HOME/vms/in_use/$vm
    mkdir -p $vm_dir
    cd $vm_dir

    echo "Copying template to VM disk..."
    cp $HOME/vms/freebsd14-cloud-init-zfs.qcow2 $vm.qcow2

    echo "Creating seed ISO..."
    cloud-localds $vm.iso ~/vms/$vm-user-data.yaml

    echo "Resizing disk..."
    qemu-img resize $vm.qcow2 50G

    echo "Launching VM..."
    virt-install \
        --connect qemu:///system \
        --name $vm \
        --memory 4096 \
        --vcpus 4 \
        --disk path=$vm.qcow2,format=qcow2,bus=virtio \
        --disk path=$vm.iso,device=cdrom \
        --os-variant freebsd14.0 \
        --import \
        --network network=maikenet,model=virtio,mac=$mac \
        --graphics spice \
        --noautoconsole

    echo "VM $vm launched."
end

# eg: createvm freebsd2 52:54:00:6c:3c:58 
 

Destroying machines quickly with Fish shell

function destroyvm
    if test (count $argv) -lt 1
        echo "Usage: destroyvm <vm-name>"
        return 1
    end

    set vm $argv[1]

    echo "Destroying VM $vm..."
    virsh -c qemu:///system destroy $vm

    echo "Undefining VM $vm..."
    virsh -c qemu:///system undefine $vm

    set disk $HOME/vms/in_use/$vm/$vm.qcow2
    set seed $HOME/vms/in_use/$vm/$vm.iso
    
    if test -f $disk
        echo "Deleting disk $disk..."
        rm -f $disk
    else
        echo "Disk $disk not found, skipping."
    end
    
    if test -f $seed
        echo "Deleting seed $seed..."
        rm -f $seed
    else
        echo "Disk $seed not found, skipping."
    end
end

Zerotier on creation with self-authorisation

This is something I'm experimenting with, installing Zerotier and joining a network are easy steps but I want it to self-authorise too. I want the variables to be fed into the cloud config so I made another fish script. This is my current $HOME/vms/user-data.yaml template from where all my machines are created.

You'll notice the addition of a script to run once the machine is up.

#cloud-config
hostname: freebsd1
users:
  - name: maikel
    shell: /usr/local/bin/fish
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: "$6$L80IKTw.......sH0"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3.....el.dev
ssh_pwauth: True
keyboard:
  layout: es
packages:
  - fish
  - sudo
  - mkpasswd
  - neovim
  - ncdu
  - zerotier
  - curl
  - jq
write_files:
  - path: /root/join-network.sh
    permissions: '0755'
    content: |
      #!/bin/sh
      NWID="111111111111111" && \
      ZT_TOKEN="222222222222" && \
      zerotier-cli join "$NWID" && \
      MEMBER_ID=$(zerotier-cli info | awk '{print $3}') && \
      curl -H "Authorization: token $ZT_TOKEN" -X POST \
        "https://api.zerotier.com/api/v1/network/$NWID/member/$MEMBER_ID" \
        --data '{"config": {"authorized": true}}'
runcmd:
  # Enable SSH
  - sysrc sshd_enable=YES
  - service sshd start

  # Set Spanish keyboard permanently
  - sysrc keymap="es.kbd"
  - service syscons restart

  # Optional: set root and maikel shells to fish explicitly
  - pw usermod root -s /usr/local/bin/fish
  - pw usermod maikel -s /usr/local/bin/fish
  
  # Auto resize main partition
  - gpart recover vtbd0
  - gpart resize -i 4 vtbd0 
  - zpool online -e zroot /dev/vtbd0p4 
  
  # Zerotier joy
  - sysrc zerotier_enable="YES"
  - service zerotier start

The function below clone_user_data VM_NAME applies the ZT_TOKEN and ZT_NW environment variables to a new user-data.yaml file with the name of the hostname.

Then once the machine is up and running you can just and simply run /root/join-network.sh as root to both join and self-authorise zerotier. I tried to do it as part of the cloud-init functions but it breaks, let alone is not something that I always need.

function clone_user_data
    if test (count $argv) -ne 1
        echo "Usage: clone_user_data <vmname>"
        return 1
    end

    set NEWVM $argv[1]
    set BASE "$HOME/vms/user-data.yaml"
    set OUT "$HOME/vms/$NEWVM-user-data.yaml"

    if not test -f $BASE
        echo "Error: base file $BASE does not exist"
        return 1
    end

    if test -z "$ZT_TOKEN"
        echo "Error: ZT_TOKEN environment variable not set"
        return 1
    end

    if test -z "$ZT_NWID"
        echo "Error: ZT_NWID environment variable not set"
        return 1
    end

    sed \
        -e "s/^hostname:.*/hostname: $NEWVM/" \
        -e "s/NWID=\"[0-9]*\"/NWID=\"$ZT_NWID\"/" \
        -e "s/ZT_TOKEN=\"[0-9]*\"/ZT_TOKEN=\"$ZT_TOKEN\"/" \
        $BASE > $OUT

    echo "Created config: $OUT"
end

# Example: clone_user_data freebsd4
# generates a file ~/vms/freebsd-user-data.yaml ready to be used by creatvm

Installing a desktop environment

I used KDE for this experiment, you really just need to read the FreeBSD Handbook and follow it step by step. I even created its own desktop-user-data.yaml file for this one in case I ever need the desktop.

The yaml file for cloudinit

#cloud-config
hostname: desktop
users:
  - name: maikel
    shell: /usr/local/bin/fish
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: "$6$L80IKTw............osH0"
    ssh_authorized_keys:
      - ssh-ed25519 ....... maikel.dev
ssh_pwauth: True
keyboard:
  layout: es
packages:
  - fish
  - sudo
  - mkpasswd
  - neovim
  - ncdu
  - xorg 
  - kde 
  - sddm
runcmd:
  # Enable SSH
  - sysrc sshd_enable=YES
  - service sshd start

  # Set Spanish keyboard permanently
  - sysrc keymap="es.kbd"
  - service syscons restart

  # Optional: set root and maikel shells to fish explicitly
  - pw usermod root -s /usr/local/bin/fish
  - pw usermod maikel -s /usr/local/bin/fish
  
  # Auto resize main partition
  - gpart recover vtbd0
  - gpart resize -i 4 vtbd0 
  - zpool online -e zroot /dev/vtbd0p4 

  # Add KDE 
  - pw groupmod video -m maikel
  - sysrc dbus_enable="YES"
  - service dbus start
  - sysctl net.local.stream.recvspace=65536
  - sysctl net.local.stream.sendspace=65536
  - sysctl -f /etc/sysctl.conf
  - sysrc sddm_enable="YES"
  - sysrc sddm_lang="es_ES"
  - setxkbmap -layout es
  - service ssdm start

The machine has a few differences, I assigned more total system memory (8GB) and hiked the RAM assigned to the video card too. My mouse is a USB one so only works with that line in input. If yours isn't USB delete that line.

virt-install \
  --connect qemu:///system \
  --name desktop \
  --memory 8192 \
  --vcpus 4 \
  --disk path=desktop.qcow2,format=qcow2,bus=virtio \
  --disk path=desktop.iso,device=cdrom \
  --os-variant freebsd14.0 \
  --import \
  --video qxl,ram=524288,vram=262144,vgamem=262144 \
  --network network=maikenet,model=virtio,mac=52:54:00:6f:3c:58 \
  --input type=mouse,bus=usb  \
  --graphics spice \
  --noautoconsole # Added this here because I prefer to let it run first. 

Cleaning up

# See the machines
sudo virsh list --all

# The first stops immediately the machine
sudo virsh destroy freebsd14

# This second removes it from the pool of VMs of libvirtd
sudo virsh undefine frebsd14

# Delete any pre-made seed just in case
rm -rf seed.iso

Some oddities

These are some painful parts from the process.

The command Virt-install and "~"

I don't know why the path can't interpret "~" hence why I did it all from the _/vms~ folder. In this version I change "~" it to $HOME in all scripts for consistency.

Run without virt-viewer

Sometimes you want to install and see nothing, in those case use

  --graphics spice \
  --noautoconsole  

At the end of the virt-install command, this runs the system with graphics enable but doesn't attach any viewer to it.