Xen

I'm not a huge fan of Xen.

dom0 can't handle too much memory

Problem: you try to boot your Xen host and it locks up during boot with a message like this:

FATAL: Error inserting dm_mod (/lib/modules/2.6.32-5-xen-amd64/kernel/drivers/md/dm-mod.ko): Cannot allocate memory
done.
Begin: Waiting for root file system ... done
Gave up waiting for root device.

The problem is that your physical machine has more memory than dom0 can handle. In my case I was working with a server with 384 GB of RAM. The solution was to set a max memory limit for the Xen hypervisor in the GRUB boot menu. The grub.cfg should have a line similar to this:

    multiboot   /xen-4.0-amd64.gz placeholder

It should be modified to something like this:

    multiboot   /xen-4.0-amd64.gz placeholder dom0_mem=512M,max:512M 

See also: http://wiki.xen.org/wiki/Xen_Best_Practices#Xen_dom0_dedicated_memory_and_preventing_dom0_memory_ballooning and http://wiki.debian.org/Xen#Other_configuration_tweaks

The exact operations you need to update grub.cfg will vary from platform to platform. On Debian 6 I did this:

dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
sed -i -e '$aGRUB_CMDLINE_XEN="dom0_mem=2048M,max:2048M"' /etc/default/grub
update-grub
sed -i -e 's/(enable-dom0-ballooning .*)/(enable-dom0-ballooning no)/' -e 's/(dom0-min-mem .*)/(dom0-min-mem 2048)/' /etc/xen/xend-config.sxp
reboot

XENDOMAINS_SAVE

Edit /etc/default/xendomains and set XENDOMAINS_SAVE to be empty. This controls the feature that allows Xen to save the guest's running state when dom0 is shutdown. I almost never need this feature. It uses a lot of disk space.

#XENDOMAINS_SAVE=/var/lib/xen/save
XENDOMAINS_SAVE=""

xend won't start

I found that this happened when my dom0 ran out of disk space. For me the solution was, "don't run out of disk space".

guests start to have erratic networking

I found that this happened when my dom0 ran low on disk space. I am not certain that this is the cause because there were no useful messages in dmesg or any other log files .