Gaming VM in a Headless Ubuntu 18.04 Server with GPU Passthrough

How to have your own private game streaming service

Benjamin Hou

11 minute read

Introduction

Being a developer, my platform of choice is first and foremost macOS. Having used Windows and macOS as my primary OS for 10 years each equally (Windows: 2001-2011 and macOS: 2009-present), I have found the UI of macOS more intuitive and easier to use, especially with Apple’s philosophy of ‘it just works’. With macOS being UNIX at heart definitely have helped with a lot of programming assignments through university. Now working in the field of deep learning, macOS is still my main platform driver. With the requirement of high-powered GPUs, my work flow adapted to a 2 step process; 1. develop deep learning models locally, 2. deploy on large scale server for training.

This means I can use my low-powered MacBook as the front end and daily driver, with a high-powered headless computer/server as the backend for training. The headless machine is a Ubuntu 18.04 LTS server with 2 GPUs installed. As the system is used specifically for computations; there are no monitors or kb/m attached, and can only be accessed through shell and SSH. A desktop environment is actually not needed, and better if it doesn’t run at all (hence the server install). In the case of doing development on the same computer as training models, and if the training process is running on the same GPU as the one that is driving the display, the UI may become slow and laggy with possible crashes as graphics resources become limited. This can be circumvented with a low-end GPU, specifically used for driving the display.

As most games are primarily developed for Windows (albeit recently more and more are being released on the Mac also), this post solves a problem for a setup that is tailored to a front-end/back-end workflow. We aim to achieve to run games in a Windows10 Virtual Machine (VM) on the back-end host with GPU passthrough. The games themselves would then be streamed to the front-end Mac using Steam In-Home Streaming.

DISCLAIMER: this post is tailored towared Intel and Nvidia hardware, as I do not own any AMD hardware. It is possible for AMD too with further googling. This guide follows closely to the 2 Reddit posts and Puget System’s post (see References).

Setup and Installation

1. Check Hardware and Configure Bios

First and foremost the hardware must support virtualisation technology and is enabled in the BIOS (Google/YouTube/Manual). Check thoroughly that all options are enabled.

  • VT-d: Intel VT for Directed I/O
  • VT-x: Intel Virtualization Technology

A monitor, keyboard and mouse would be required just for setting things up.

2. Install Necessary Software

sudo apt-get update
sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker

3. Update GRUB and kernel modules

For Ubuntu to load IOMMU properly, intel_iommu=on needs to be added to the boot flag. sudo nano /etc/default/grub and append intel_iommu=on to the line with GRUB_CMDLINE_LINUX_DEFAULT. Don’t forget to sudo update-grub.

Example cat /etc/default/grub:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="nomodeset pci=nomsi intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Edit /etc/modules to add specific kernel modules so that they are loaded during boot.

Example cat /etc/modules:

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

pci_stub # probably not needed
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

Reboot machine

4. Check Virtualisation and IOMMU groupings

Check that virtualisation is enabled by running kvm-ok

farrell@skylab:~$ kvm-ok 
INFO: /dev/kvm exists
KVM acceleration can be used

Check that IOMMU is enabed by running dmesg | grep -e DMAR -e IOMMU.

farrell@skylab:~$ dmesg | grep -e DMAR -e IOMMU
...
[    0.000000] DMAR: IOMMU enabled
...

From here, this shell snippet checks IOMMU groupings. It is critical that the GPUs are in separate IOMMU groups, otherwise it’s not going to work!

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); \
do echo "IOMMU group $(basename "$iommu_group")"; \
for device in $(ls -1 "$iommu_group"/devices/); \
do echo -n $'\t'; lspci -nns "$device"; done; done

Example output:

farrell@skylab:~$ for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); \
> do echo "IOMMU group $(basename "$iommu_group")"; \
> for device in $(ls -1 "$iommu_group"/devices/); \
> do echo -n $'\t'; lspci -nns "$device"; done; done
...
IOMMU group 31
	02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b00] (rev a1)
	02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10ef] (rev a1)
IOMMU group 32
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b00] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10ef] (rev a1)
...
farrell@skylab:~$

Proceed if these checks are OK.

5. GPU bind scripts

Previous tutorials would require a step where a dummy driver (pci-stub) claims access to the GPU before the Nvidia driver claims it. This is step is not necessary as there is no display server running, and the GPU can be unbound easily.

Create 2 scripts; one to bind the GPU to VFIO, and one to bind the GPU back to Nvidia.

Create gpu-bind-vfio.sh :

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

and gpu-bind-nvidia.sh:

#!/bin/bash

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/nvidia/new_id
done
  • Run sudo ./gpu-bind-vfio.sh 0000:02:00.0 0000:02:00.1 to detach the GPU from the Nvidia driver and binding it to VFIO.
  • Run sudo ./gpu-bind-nvidia.sh 0000:02:00.0 0000:02:00.1 to detach the GPU from the VFIO driver and binding it to Nvidia.

Verify the binding by running lspci -nnk:

Example output:

farrell@skylab:~$ lspci -nnk
...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b00] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:119a]
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10ef] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:119a]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b00] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:119a]
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10ef] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:119a]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
...

6. BIOS Setup

Pick either UEFI or Legacy BIOS for qemu. However, the latest version of qemu seems to have trouble with booting the Windows 10 install iso via TianoCore UEFI BIOS.

6a. UEFI BIOS

Download the UEFI BIOS file from here and save the file OVMF.fd at a convenient place.

6b. Legacy BIOS

Download SeaBIOS from here and build in tree.

git clone https://github.com/qemu/seabios.git
cd seabios
make

The built BIOS will be in out/bios.bin folder.

7. Install ISO and SCSI drivers

Download Windows 10 iso from here.

Download the VFIO drivers for Windows from here. The stable option should suffice, at time of writing it’s virtio-win-0.1.141.iso.

Save the filed to a convenient place.

8. HDD image

Create a 120GB dynamic size HDD win10_120G_HDD.img

qemu-img create -f qcow2 -o preallocation=metadata,compat=1.1,lazy_refcounts=on win10_120G_HDD.img 120G

9. Install Boot Script

Connect keyboard and mouse to the host computer and the monitor to the GPU being passed through.

Finding keyboard and mouse device IDs by running lsusb. In my case it’s 03f0:0024 for the keyboard and 093a:2510 for the mouse. Make a note of these IDs.

farrell@skylab:~$ lsusb
...
Bus 003 Device 012: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 003 Device 008: ID 03f0:0024 Hewlett-Packard KU-0316 Keyboard
...

The directory should look like this:

farrell@skylab:~/Windows10$ tree
.
├── gpu-bind-nvidia.sh
├── gpu-bind-vfio.sh
├── install
│   ├── Win10_1709_English_x64.iso
│   └── virtio-win-0.1.141.iso
├── seabios
│   ├── ...
│   ├── out
│   │   ├── ...
│   │   ├── bios.bin
│   │   └── ...
└── win10_120G_HDD.img

Create a boot script for the install step, nano win10_install.sh:

#!/bin/bash

qemu-system-x86_64 \
  -enable-kvm \
  -m 32768 \
  -cpu host,kvm=off \
  -smp cores=4,threads=2 \
  -vga none \
  -bios seabios/out/bios.bin \
  -device qemu-xhci,id=usb,bus=pci.0,addr=0x4 \
  -device usb-host,vendorid=0x03f0,productid=0x0024 \
  -device usb-host,vendorid=0x093a,productid=0x2510 \
  -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=02:00.1 \
  -device virtio-scsi-pci,id=scsi \
  -drive file=install/Win10_1709_English_x64.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=install/virtio-win-0.1.141.iso,id=virtiocd,if=none,format=raw -device ide-cd,bus=ide.1,drive=virtiocd \
  -drive file=win10_120G_HDD.img,id=disk,format=qcow2,if=none,cache=writeback -device scsi-hd,drive=disk \
  -net none \
  -nographic
  • -enable-kvm: Enable KVM full virtualization support.
  • -m 32768: 32GB RAM
  • -cpu host,kvm=off: use host CPU profile and hide KVM signature
  • -smp cores=4,threads=2: Hyperthreaded Quad Core CPU (equivalent to an i7)
  • -vga none: do not emulate VGA card
  • -bios seabios/out/bios.bin: Legacy BIOS file
  • -device qemu-xhci,...: qemu xHCI USB bus
  • -device usb-host,vendorid=0x03f0,productid=0x0024: Keyboard
  • -device usb-host,vendorid=0x093a,productid=0x2510: Mouse
  • -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on: Passthrough GPU
  • -device vfio-pci,host=02:00.1: Passthrough GPU Audio
  • -device virtio-scsi-pci,id=scsi: VirtIO SCSI PCI bus
  • -drive file=install/Win10_1709_English_x64.iso,...: Windows10 install iso
  • -drive file=install/virtio-win-0.1.141.iso,...: VirtIO drivers for Windows
  • -drive file=win10_120G_HDD.img,...: Virtual HDD
  • -net none: disabled NIC
  • -nographic: since we are accessing the host via SSH, the qemu graphics window should be disabled

Networking does not need to be configured here, default qemu configuration configures the VM to share the internet with the host (i.e. the host provides NAT for network traffic from the VM). Here, the NIC is disabled for simplicity.

Post Install

1. Bridged networking

For Steam in-home streaming to work, the VM needs to be bridged to the router so they client machine and host machine are on the same subnet. This relies on the ‘Public Bridge’ section of the tutorial found here. As the machine is accessed via SSH, configuration needs to be done carefully as it’s very easy to lose access to the host when the networking service is restarted. The following steps will be a combination of Solution 1 and Solution 2.

Edit networking profile so that a bridged interface is used sudo nano /etc/network/interfaces

Example modification:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eno1
#iface eno1 inet dhcp

auto br0
iface br0 inet dhcp
bridge_ports eno1

Restart networking service

sudo service networking restart

Make a backup of the qemu network interface init script

sudo cp -v /etc/qemu-ifup /etc/qemu-ifup.bak
sudo nano /etc/qemu-ifup

Use the script provided in Solution 2, save as qemu-ifup

#!/bin/sh
set -x

switch=br0

if [ -n "$1" ];then
        #tunctl -u `whoami` -t $1
        ip tuntap add $1 mode tap user `whoami`
        ip link set $1 up
        sleep 0.5s
        #brctl addif $switch $1
        ip link set $1 master $switch
        exit 0
else
        echo "Error: no interface specified"
        exit 1
fi

2. Run Boot Script

Generate a MAC address for the VM NIC using bash

printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

VM boot script can now be modified to run state. Remove the Windows 10 install media iso, as well as the VirIO drivers iso. Add line -device e1000,netdev=net0,mac=FAKE_MAC_ADDRESS -netdev tap,id=net0 so that networking is now bridged instead of NAT.

Create a boot script for the launching the VM, nano win10_run.sh:

#!/bin/bash

qemu-system-x86_64 \
  -enable-kvm \
  -m 32768 \
  -cpu host,kvm=off \
  -smp cores=4,threads=2 \
  -vga none \
  -bios seabios/out/bios.bin \
  -device qemu-xhci,id=usb,bus=pci.0,addr=0x4 \
  -device usb-host,vendorid=0x03f0,productid=0x0024 \
  -device usb-host,vendorid=0x093a,productid=0x2510 \
  -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=02:00.1 \
  -device virtio-scsi-pci,id=scsi \
  -drive file=win10_120G_HDD.img,id=disk,format=qcow2,if=none,cache=writeback -device scsi-hd,drive=disk \
  -device e1000,netdev=net0,mac=DE:AD:BE:EF:B5:40 -netdev tap,id=net0 \
  -nographic

3. Install Software

Install the latest patches from Microsoft and GPU driver from Nvidia.

4. Remote Desktop Access

Enable remote access via this tutorial here. After this, the keyboard, mouse and monitor can be removed.

5. Disable Power Management Sleep

Can’t wake the VM if it enters sleep state!

Synthetic Benchmarks

Host:
* CPU:          Broadwell-E i7 
* GPU:          Titan X (Pascal)
* OS:           Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-33-generic x86_64)
* GPU Driver:   390.87

Virtual Machine:
* CPU:          4 Cores, 2 Threads
* CPU:          Titan X (Pascal) VFIO Passthrough
* Memory:       32GB
* OS:           Windows 10 Pro (Version 1709, OS Build 16299.611)
* GPU Driver:   399.07

1. CINEBENCH R15

OpenGL:   155.43 fps
CPU:      1189 cb

2. Unigine Heaven (Extreme HD 1080p Preset)

FPS:      154.3
Score:    3886
Min FPS:  9.5
Max FPS:  332.6

3. Unigine Valley (Extreme HD 1080p Preset)

FPS:      95.4
Score:    3991
Min FPS:  21.4
Max FPS:  177.6

4. Unigine Superposition (Extreme HD 1080p Preset)

FPS:      41.75
Score:    5582
Min FPS:  33.02
Max FPS:  51.83

References