Fedora virtualization setup as Xen dom0

/ 25 March, 2014

In my previous blog post about choosing the right dom0, I came to the conclusion that in the face of an upcoming RHEL7 which is based on Fedora 19/20, it made sense to set up my dom0 for using Fedora 20.

Given the similarity between Fedora 20 and RHEL7, it should be the same process for RHEL, except for the fact that RHEL7 beta has dom0 support in the kernel turned off, according to this post on the Centos mailing list.  I’m hoping that the Centos people will see fit to provide a centos-plus kernel, or even turn on dom0 support in the default kernel, given that its already in the codebase.  What I’d really like is for RHEL7 customers to ask Redhat to enable dom0 support!

So anyway, I digress.  Here are the steps I went through to set up a working dom0 on Fedora 20.  It wasn’t all smooth sailing for me, as I’m very used to Centos6 (and earlier), so the change to using the systemctl init system was a significant one.  Its worth noting that my hardware is the following:

  • Xeon X3450 (quad core w/ hyperthreading)
  • Supermicro X8SIL-F (with 2 intel NIC’s onboard)
  • 32GB RAM
  • Intel X25-M 160GB SSD
  • LSI 9211.8 HBA card (actually a Dell Perc H310 reflashed with LSI firmware)
  • Dual port Intel PRO/1000 PT PCIe nic

I bring it up partly because hey, I like having “real” server hardware for the first time in um.. ever, and partly because you might have some minor variations to the rest of these notes depending on controller cards, network chipsets etc.

First up – installation.. here’s the basics I went with.

  • 500 MB /boot partition (using ext4)
  • LVM disk with
    • 10gigabyte root partition (/)
    • 4 gigabyte swap partition
    • Lots of free space for VM’s
  • Then selected the following software packages:
    • XFCE Desktop  (because sometimes its nice to have a vnc desktop off the dom0 – mostly for using virt-manager)
    • Uncheck the Multimedia and Applications
    • Check the “System Tools” and “Administrative Tools”

Once the installer has finished and you’ve rebooted the box, its time to make a few changes to the system to facilitate Xen.

First up, disabling the firewall, so that VNC will work (note – my dom0 sits behind a pfsense firewall and is not accessible from an external network):

systemctl stop firewalld.service
systemctl disable firewalld.service

Yep, that’s right, no more “service firewall stop” or “chkconfig firewall off”.. welcome to the future!

Then disable that old chestnut, selinux:

vi /etc/selinux/config

Change the line starting with “SELINUX=” to:

SELINUX=disabled

Once that’s done, update the system packages:

yum update -y

And install xen, as we’ll probably need it 🙂

yum install xen

We also need to configure grub2 to load the xen kernel.. yes, more unfamiliar stuff from the older model.  However, before we do that, here’s a small modification which improved performance and stability for me by taking away the need to balloon the dom0 memory:

vi /etc/default/grub
add this line:
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=min:1024M,max:1024M"

And now lets run the commands to update the grub2 config file, and then set the default boot entry:

grub2-mkconfig -o /boot/grub2/grub.cfg
grep ^menuentry /boot/grub2/grub.cfg | cut -d "'" -f2
grub2-set-default <menu entry title you want>  (this one:  "Fedora, with Xen hypervisor")

Next, disable the infernal NetworkManager service, and go with networking which is more xen-friendly.

systemctl disable NetworkManager.service || systemctl restart network.service
chkconfig network on
Having turned the network service on, lets sort out a bridge interface for use by xen domU’s.
Start by making a config file for the bridge – I like to call it “xenbr0” to avoid confusion.
vi /etc/sysconfig/network-scripts/ifcfg-xenbr0
And use these contents for it:  (note this assumes a dhcp server is on your network – alternatively you can specify BOOTPROTO=none, along with the ip address, gateway address, etc.)
DEVICE=xenbr0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no
Then modify the existing interface to turn it into a bridge:
vi /etc/sysconfig/network-scripts/ifcfg-em1
And here are the additions:
IPV6INIT="no"     <-- I like to disable IPv6 as I don't use it (yet)
BRIDGE=xenbr0     <-- this is the important line
NM_CONTROLLED=no  <-- this is handy in case NetworkManager rises from the dead like the vampire it is
And restart the network to make sure all is hunky-dory
systemctl restart network.service
Last thing to do before reboot is to tell the box not to start XFCE – I like to have XFCE around for vncserver purposes, but I prefer a text mode console on the box.  AGain, this has changed a lot due to the use of systemd, so lets update the default boot runlevel using the new systemd format
ln -sf /lib/systemd/system/multi-user.target /etc/systemd/system/default.target
Ok, I lied, a few more things occurred to me to add before the reboot.. not all mandatory or even necessary, but I like them.
yum install -y libvirt virt-manager ntp
chkconfig ntpd on
systemctl start ntpd.service
And finally… trumpet fanfare:
reboot
After a reboot, lets do a quick check to make sure xen is running (in case you blinked and missed the XEN part of system out during boot):
xl info
Success!  Xen is now working, so if thats all you wanted to achieve, then time to start installing a domU, and ignoring the rest of this post.
Not me however – I like to make shit complicated.. usually it ends up pushing the boundaries of what the software will support, or hardware issues, but hey, I like to have it all “just so” and in this case, it means witholding hardware to pass through to my NAS virtual machine (LSI disk controller with a bunch of WD Red drives attached), and firewall vm (pair of dedicated NIC’s for external and internal network interfaces).
Lets tell xen we want to reserve hardware, by using the xen-pciback setup to hide the various devices (HBA and 2 NIC’s) for PCI passthru.  Start by creating a conf file to hold the relevant xen entries:
vi /etc/modprobe.d/xen.conf

And use contents similar to this in it:

options xen-pciback hide=(02:00.0)(03:00.0)(03:00.1)(05:00.0)
install e1000e /sbin/modprobe xen-pciback ; /sbin/modprobe --first-time --ignore-install e1000e
install mpt2sas /sbin/modprobe xen-pciback ; /sbin/modprobe --first-time --ignore-install mpt2sas
# left from old hardware
#install r8169 /sbin/modprobe xen-pciback ; /sbin/modprobe --first-time --ignore-install r8169
 Due to a bug in how the mpt2sas driver is loaded, I also found it necessary to add lines to the end of this file:
vi /etc/sysconfig/modules/xen.modules
The additions are to unload the mpt2sas driver, unload the xen-pciback driver, and reload the xen-pciback driver (which will now pick up the LSI card).  Add these lines at the end of the xen/modules file:
rmmod mpt2sas
rmmod xen-pciback
modprobe xen-pciback

Last round of tests.. first make sure the devices are withheld:

xl pci-assignable-list

Should now return the devices which are reserved for pci passthrough.  Lastly, reboot the box, just to make sure it all comes back up with reserved devices etc.  And all done!  Well ok, who am I kidding.. now the really hard work begins, installing, configuring and testing the various VM’s that I’ll be expecting to run 24/7 for a couple of years with minimal interruptions.

Ooroo!

 

Related posts

4 thoughts on “Fedora virtualization setup as Xen dom0”

Leave a Reply

Your email address will not be published. Required fields are marked *