GPU passthrough for bhyve on FreeBSD 14.0-RELEASE and 14.1

21-05-2024

Table of Contents

We recently purchased a GPU for our University computer club [1] for Machine Learning tasks. Being most comfortable on a FreeBSD system I decided that running a VM through bhyve would be most suitable. Especially as I expect that several students will connect to this machine concurrently using harmful software such as Visual Studio Codes remote development over SSH extension.

Why even bother

From experience I know that VSCode launches a gazillion processes and wont kindly terminate them. It also downloads binaries from microsoft servers and runs a little node session remote on the server for each user and doesnt reuse old ones. A bonus is that it doesnt run naively on FreeBSD (You can get this working using linuxulator [2] but dont bother).

But the problem is that meanwhile Bhyve does support PCI-e passthrough its GPU support is lacking according to people online. [3]

bhyve supports passing of host PCI devices to a virtual machine for its exclusive use of them.
Note: VGA / GPU pass-through devices are not currently supported.

That last part is not really true, the problem lies within some drivers (NVIDIA) only supporting hypervisors with the KVM signature, so we need to patch Bhyve.

Luckily patches are already available [4], and the instructions are as simple as following the FreeBSD developers handbook [5].

Lets get started!

First get the sources for FreeBSD, you probably only care about the branch with the patches applied. They reside on github from the Beckhoff github account under the tag phab/corvink/14.0/nvidia-wip.

cd /where/you/want/to/build
git clone --depth 1 --branch phab/corvink/14.0/nvidia-wip https://github.com/Beckhoff/freebsd-src

buildkernel

Now we can either use the script provided by corvink@ [6] or just follow the handbook.

cd freebsd-src
make -j64 buildkernel
make installkernel

Now the new kernel is installed, the previous kernel is located at /boot/kernel.old/kernel

make bhyve

Now its time to build bhyve with our patches, this will go quick. The provided script automates this for you. (really, use the script [6])

cd include
make -j64
make install  
cd lib/libvmmapi
make -j64
make install  
cd sys/modules/vmm
make -j64
make install  
cd usr.sbin/bhyve
make -j64
make install  
cd usr.sbin/bhyvectl
make -j64
make install  
cd usr.sbin/bhyveload
make -j64
make install

Really, just use the script. But now you should be ready to reboot into your freshly patches system!

For future FreeBSD releases you can simply go in and apply the patch manually if necessary.

Bhyve passthrough

Now to have our pci device to passthrough to bhyve we first figure out what id its assigned. This is basically following the wiki page at [3].

$ pciconf -v -l | grep -B 1 -A 3 NVIDIA  
ppt0@pci0:193:0:0:      class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2204 subvendor=0x1028 subdevice=0x3880
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 [GeForce RTX 3090]'
    class      = display
    subclass   = VGA
ppt1@pci0:193:0:1:      class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1aef subvendor=0x1028 subdevice=0x3880
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA

Now note down those numbers and insert them into /boot/loader.conf as following

# echo pptdevs="193/0/0 193/0/1" >> /boot/loader.conf

Now the host wont steal them when the system is started, now to map them into Bhyve. If you want to passthrough another device, such as an USB hub to the VM you can specify more devices by appending N to pptdevs such as pptdevs2="194/0/0".

We use the excellent bhyve-vm [7] utility to simplify vm management.

Simply add the following to the end of your $VM.conf file, they will now show up in pci slots 9:0 and 9:1 on the VM. You might want to make sure that you dont make them to a spot thats already occupied, if so just go higher.

passthru0="193/0/0=9:0"
passthru1="193/0/1=9:1"

And presto!

Now you can reboot your FreeBSD host and your VM will have access to the GPU.

On a linux VM you can verify that the patches were applied correctly by checking your hypervisor signator by running dmesg

# dmesg | grep KVM

Update for 14.1

The patch still works on 14.1 it just needed some tiny refactoring as read_config was renamed to pci_host_read_config. A patch file is provided nvidia.patch. Apply it by running git apply -v nvidia.patch after cloning 14.1 by running git clone --depth 1 --branch releng/14.1 https://github.com/freebsd/freebsd-src

References

[1] https://df.lth.se/
[2] https://wiki.freebsd.org/Linuxulator
[3] https://wiki.freebsd.org/bhyve/pci_passthru
[4] https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip
[5] https://docs.freebsd.org/en/books/developers-handbook/kernelbuild/
[6] http://okturing.com/src/19931/body
[7] https://github.com/churchers/vm-bhyve