I will attempt to take this step by step since it’s a complicated process if you’re a first timer on more advanced things in linux. Taking it step by step means that there will be many steps but most of them will be very short and thoroughly explained. I will be making some assumptions since they are basically the foundation of having a good experience when doing this:
- your graphics card supports UEFI
- you are running a kernel that is newer than 4.1
- you are capable of swapping your specific details for mine where needed
- you are running arch linux or a distribution based on it if you follow my guide
- you fulfill the hardware requirements
some introductory words:
Wouldn’t it be amazing if you could use one tower computer as two or more desktops? Of course it would be. There are many technologies in the enterprise such as SRIOV but what about the desktop? What can you do then? The internet (or at least some small corners of it) have been taken by storm since the release of linux kernel version 4.1 and later. Terms like VFIO, QEMU, IOMMU, GPU passthrough, PCI device separation and many more have been associated with the subject. You can read all of that on your own time though which is what I advise you do. I will put some links to resources such as the arch wiki page about this which is the starting point fro many people who are interested in this sort of thing. So in simple terms, what is this all about? The idea is this: you take a graphics card plugged in your computer and let it be used by a virtual machine. The host operating system can’t use it, in my case linux but the guest operating system can, in my case windows. Instead of waxing philosophical I decided to go for this configuration to showcase a common practical application of all of these technologies.
On to the actual steps now.
Step 0: Hardware
Your cpu should support VT-d if Intel or AMD-Vi if AMD. Your motherboard should have good IOMMU grouping (there is a workaround if it doesn’t, see the arch wiki page I will link in the end for more information). You have a monitor with multiple inputs or two monitors or a KVM switch. You have two sets of mouse and keyboard.
Step 1: BIOS
Two categories here:
- you are using intel integrated graphics for the host
- you are using a dedicated gpu for the host
If you are in group 1 you need to modify your bios so the output upon boot is the iGPU and enable VT-d. If you are in group 2 you need to modify your bios so the output upon boot is the gpu that will be used for the host and enable VT-d or AMD-Vi, whatever is applicable to you.
Step 2: GRUB
You need to edit /etc/default/grub by adding the option intel_iommu=on or amd_iommu=on, depending on your cpu vendor, within the quotes of GRUB_CMDLINE_LINUX_DEFAULT. After that, update grub using the command grub-mkconfig -o /boot/grub/grub.cfg and then reboot.
Step 3: Checking
First, run dmesg|grep -e DMAR -e IOMMU and you should see something like:
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)
[ 0.000000] Intel-IOMMU: enabled
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1
[ 0.536212] DMAR: No ATSR found
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation
[ 0.536231] IOMMU: Setting RMRR:
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 2.182790] [drm] DMAR active, disabling use of stolen memory
If you see something like that, you’re on a good path. To further ensure that your IOMMU groups are good, run the script below (you have to make a new file with nano, paste the script and then make it executable with chmod +x):
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf ‘IOMMU Group %s ‘ “$n”
lspci -nns “${d##*/}”
done;
To quote the arch wiki:
“Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.”
“This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks.”
In my case I had a PCI bridge in the same IOMMU group as my video card and the audio device associated with it so it was fine.
Step 4: VFIO IDs
You need to run lspci -nn and find the VID:PID that corresponds with your gpu and your gpu’s audio device if it has one. Mine are these: 10de:1004 10de:0e1a (gpu and gpu audio device respecitively). Afterwards just take those two IDs and put them in /etc/modprobe.d/vfio.conf like this: options vfio-pci ids=10de:1004,10de:0e1a (make sure to replace my IDs with yours).
Step 5: Modules
In /etc/mkinitcpio.conf check that you have (in this order) MODULES=”vfio vfio_iommu_type1 vfio_pci vfio_virqfd” and HOOKS=”modconf”. Following that, you just have to run mkinitcpio -p linux in order to regenerate the initramfs and then reboot. WARNING: you will not be able to get output from your gpu after the machine has rebooted since the gpu will be claimed by vfio and not by the driver it was previously using, you will only get output from it when it’s been assigned to the VM.
Step 6: Checking once more
Run dmesg | grep -i vfio to see if your gpu and it’s audio device are being claimed by vfio. Sometimes it doesn’t show up correctly there so in order to double-check just run lspci -nnk -d 10de:1004 and lspci -nnk -d 10de:0e1a (replacing my IDs with yours as usual).
Step 7: Packages
To install the things necessary for this, use pacman -S qemu libvirt ovmf virt-manager firewalld.
Step 8: Config
The configuration of /etc/libvirt/qemu.conf can prove to be a little intimidating. What you need to do:
- add this:
nvram = [
“/usr/share/ovmf/ovmf_code_x64.bin:/usr/share/ovmf/ovmf_vars_x64.bin”
]
(it doesn’t show correctly but before the second line there should be four spaces)
to the end of the file - change clear_emulator_capabilities from 1 to 0,
- change the permissions to be: user = “root” group = “kvm” (exactly as I have written them)
and that’s about it for the configuration of that file. The nvram is where our OVMF files are. OVMF is basically our UEFI firmware since we don’t want to be lefte behind using legacy booting ways.
Step 9: Startup thingies
sudo systemctl enable –now libvirtd
sudo systemctl enable virtlogd.socket
sudo systemctl enable firewalld
sudo usermod -a G kvm angle (replace angle with the name of your user)
Step 10: ISO directory
I made a directory, /pub, and did a nifty chmod 777 /pub so it was surely accessible because having the iso in my home directory gave some errors.
Step 11: Drivers
We are going to be using VirtIO drivers for some things because performance is better so we need to get those drivers. Simply head to Index of /groups/virt/virtio-win/direct-downloads/archive-virtio and get the latest ones.
(I will edit and add the gui setup and stuff later)
you’re still with me? great. things are going to get easier and more gui-oriented
Step 12: VM initial creation and windows installation
Let’s open virt-manager and click the button on the top left to create a new vm and choose ISO then hit next
Click browse then browse local and go to the /pub directory where you should have your windows iso and your virtio drivers iso
then accept the default memory and cpu, we’l change it later anyway
put whatever you want in the storage here, we’ll change that too
then set things up like this so you get an IP address from your router and not end up in a double-NAT situation
okay, now to set up our vm further. I will put the screenshots and add the commentary all at once afterwards
so what did I do?
- changed from BIOS to the UEFI firmware we added to qemu.conf (that was the nvram thing)
- changed cpu topology (put 4 cores in current allocation, not 1 as I did, it was just a little mistake)
- deleted the IDE cdrom and hard drive in favor of two sata cdroms with the windows and the virtio drivers iso as well as a virtio hard drive
- I changed the boot order so it goes from the cdrom with windows to the virtio hard drive that will have windows installed and then the virtio iso (you don’t need to click the checkbox since it doesn’t matter that much because it will boot either the windows iso or the windows drive before it ever reached the virtio iso)
Step 13: windows installation
This should be the easiest step but with a little snag. You’ll have to load the virtio drivers so at install time the installer can see the virtio driver we’ve set up. so you start the vm, and at some point you get here:
you load the drivers by navigating to the right forlder (do it according to the version you’re using) and then click OK
then just click on the driver and hit next and you should see your drive
then just complete your windows installation and boot into it to finish setup and we should be just about done with that part(some extra driver installation will be needed afterwards but it’s easy mode)
Step 14: GPU and USB assignment
Let’s turn off the vm and go change some stuff so your gpu will start doing something. But first!! change your nic to virtio (you won’t have a network connection until you install the drivers for it though, I’l show you how to do that later)
now add your GPU and change the boot menu if you want
As you see from the above screenshots and the one below I removed some hardware. Namely: display spice, video qxl, serial 1, channel spice and tablet.
Now, before you get trigger happy and launch the vm, make sure to assign a mouse and keyboard so you can control the damn thing. I added a mouse/keyboard combo that has one logo (Logitech MK220).
Step 15: Post boot
So now you should be in the vm and be getting output from your GPU however you have chosen to get that output (second monitor, other output on the same monitor or KVM switch). Next we go to device manager and see what is missing drivers. Click on each thing with the warning sign and then update driver, browse to the virtio disk and don’t select anything specific just click on the CD drive, it’ll find things on it’s own now that we’re booted. That’s what you will do to get your network working as well as anything else.
Step 16: Nvidia fix
Obviously only applicable if you have an Nvidia GPU. What the deal is? Nvidia doesn’t want you to do what you can do with your hardware and needs to be slapped. So we will mask the fact that we’re in a virtual environment. If you named your vm win10 as I did you can just copy the command. Turn off the vm and then open a terminal. Type sudo EDITOR=nano virsh edit win10 which will open a big scary file. Now navigate to a segment of code that looks almost like the picture below and modify it to look as such:
You’re now ready to turn your windows vm back on, download your graphics drivers and install them with no problems.
some sources I used:
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
https://anlhzexperiences.wordpress.com/
https://forum.level1techs.com/t/play-games-in-windows-on-linux-pci-passthrough-quick-guide/108981
https://problemsolved-os.blogspot.gr/2017/03/error-failed-to-load-preset.html
https://www.reddit.com/r/VFIO/ (check out their discord server as well)
https://level1techs.com/article/ryzen-gpu-passthrough-setup-guide-fedora-26-windows-gaming-linux