QEMU/KVM with GPU Passthrough, Kernel Recompile: The Debian Way | edencomputing.com

Once your host OS has been installed and updated, it’s time to make a kernel compatible with KVM’s more advanced features. Most of these features are already available as modules in the Debian Jessie kernel, however as many others have, I found it was necessary to compile KVM support directly into the kernel for things to work smoothly with PCI passthrough.

Kernel recompile, “the Debian way”

You can use any compilation and installation method you wish, but this method will give you a set of KVM-ready .deb files which you can install on other (similar) systems. I do not intend this to be a definitive reference for compiling kernels; refer to Debian’s official Compiling a New Kernel guide.

Again this guide assumes that you have root access. This is really not the best way to compile the kernel if you’re concerned about security or not hosing the system you’re compiling on. Refer to the guide to see how to use fakeroot to compile instead.

Getting and Uncompressing the Source

Any recent kernel should do, but if you wish to duplicate my experience, and especially if you’re using the same hardware, it may be best to fetch the same kernel version I have. Grab and unpack the vanilla kernel from :

curl https://www./pub/linux/kernel/v3.x/linux-3.13.3.tar.gz  > /usr/src/linux-3.13.3.tar.gz
cd /usr/src
tar -xzf linux-3.13.3.tar.gz
ln -s linux-3.13.3 linux
cd linux

Configuring the Kernel

To configure the vanilla kernel with the same options as the Debian default kernel, simply do:
make oldconfig

Accept the default configuration options if any are presented. Now to configure the kernel, do:
make menuconfig

Change the following kernel options and modules to be compiled into the kernel (set them to [*] instead of [M]):

• Virtualization
  (CONFIG_VIRTUALIZATION=y)
• Virtualization > Kernel-based Virtual Machine (KVM) Support
  (CONFIG_KVM=y)
• Virtualization > KVM for <whichever processor you have>
  (CONFIG_KVM_INTEL=y and/or CONFIG_KVM_AMD=y)
• Bus Options (PCI etc) > PCI Stub Driver
  (CONFIG_PCI_STUB=y)
• Device Drivers > VFIO Non-Privileged userspace driver framework
  (CONFIG_VFIO=y)
• Device Drivers > VFIO Non-Privileged userspace driver framework > VFIO support for PCI Devices
  (CONFIG_VFIO_PCI=y)
• Device Drivers > VFIO Non-Privileged userspace driver framework > VFIO support for VGA Devices
  (CONFIG_VFIO_PCI_VGA=y)

The rest of these are optional (but recommended) performance enhancements you may compile as modules [M]:

• Virtualization > Host kernel accelerator for virtio net
  (CONFIG_VHOST_NET=m)
• Device Drivers > Virtio drivers > PCI driver for virtio devices
  (CONFIG_VIRTIO_PCI=m)
• Device Drivers > Virtio drivers > Virtio balloon driver
  (CONFIG_VIRTIO_BALLOON=m)

And finally these options will enhance performance and responsiveness of your virtual machines:

• Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
  (CONFIG_PREEMPT=y)
• Processor Type and Features > Timer Frequency > 1000 Hz
  (CONFIG_HZ_1000=y)

Make any other changes you need for your hardware, if any. If the default Debian configuration works you probably don’t need to change anything else. Exit, and accept the prompt to save the kernel configuration.

Compiling and creating the Debian package

In Debian, this is one simple step:
# make-kpkg --initrd --append-to-version=kvm.1 kernel_image kernel_headers

Note the option –append-to-version – this string will be added to the end of your version to differentiate this kernel from others you install or compile. Set this to something unique.

Install

Hop up a directory with cd .., and you should see several .deb files created by the build. Install the kernel image and headers using dpkg:
# dpkg -i linux-image-3.13.3-kvm.1_3.13.3-kvm1-10.00.Custom_amd64.deb linux-headers-3.13.3-kvm.1_3.13.3-kvm.1-10.00.Custom_amd64.deb

Further Kernel Configuration for Passthrough

It’s likely you compiled the radeon/noveau driver module during the last steps; if you have an AMD/ATI card that you are planning to use for PCI passthrough, you should blacklist the radeon driver to prevent it being loaded at boot and taking over the card:

# echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

IOMMU support must be enabled in the kernel at boot time to be utilized, but we haven’t configured it to do so yet.  Assuming you’ve compiled KVM support into the kernel, this should be as simple as adding “iommu=1″ to your kernel arguments.  Edit /etc/default/grub and find the line starting GRUB_CMDLINE_LINUX_DEFAULT. Simply append the option to this line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=1"

Depending on your hardware, there are likely other boot parameters you will want to use. Unfortunately this seems to be a case-by-case, trial and error endeavor to find your perfect set.  Here’s mine; these options may or may not all be necessary but they work for me:

quiet iommu=1 iommu=pt kvm.ignore_msrs=1 nomodeset pci-stub.ids=1002:6798,1002:aaa0

iommu=pt enables “passthrough” for AMD chips.  The equivalent option for Intel chipsets is intel_iommu=on.

kvm.ignore_msrs=1 prevents a bug with message handling that, reading more into it just now, I don’t think I actually need.  I believe I simply picked this up from the ArchLinux KVM/Passthrough guide.  I’d leave this off unless you have problems that lead you to specifically enable it, and disabling it seems to make no difference for me.

nomodeset I added to prevent Debian’s init scripts from trying a higher resolution console on the host, wherein it might bind the radeon driver to the primary GPU.  Since I will almost never see the pretty virtual consoles anyway, it’s an easy call.  This also shaves a second or two off boot time as it prevents scanning for all the supported monitor modes.

The pci-stub entry I will explain in the next post in this series.

Now, update grub to add these parameters:

# update-grub
# reboot


When the machine comes back up, login and check if IOMMU support is enabled and working:
dmesg | grep AMD-Vi

If things are working, you should see something like:

[ 1.065881] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[ 1.065882] AMD-Vi: Interrupt remapping enabled
[ 1.073946] AMD-Vi: Lazy IO/TLB flushing enabled


Congrats! Now you have a PCI-Passthrough compatible kernel. Next I’ll show how to reserve devices at boot time with the pci-stub driver and bind them to the VFIO driver to be used by virtual machines.