Xen

From Noah.org
Revision as of 20:55, 27 November 2012 by Root (talk | contribs)
Jump to navigationJump to search


I'm not a huge fan of Xen.

dom0 can't handle too much memory

Problem: you try to boot your Xen host and it locks up during boot with a message like this:

FATAL: Error inserting dm_mod (/lib/modules/2.6.32-5-xen-amd64/kernel/drivers/md/dm-mod.ko): Cannot allocate memory
done.
Begin: Waiting for root file system ... done
Gave up waiting for root device.

The problem is that your physical machine has more memory than dom0 can handle. In my case I was working with a server with 384 GB of RAM. The solution was to set a max memory limit for the Xen hypervisor in the GRUB boot menu. The grub.cfg should have a line similar to this:

    multiboot   /xen-4.0-amd64.gz placeholder

It should be modified to something like this:

    multiboot   /xen-4.0-amd64.gz placeholder dom0_mem=512M,max:512M 

See also: http://wiki.xen.org/wiki/Xen_Best_Practices#Xen_dom0_dedicated_memory_and_preventing_dom0_memory_ballooning and http://wiki.debian.org/Xen#Other_configuration_tweaks

The exact operations you need to update grub.cfg will vary from platform to platform. On Debian 6 I did this:

dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
sed -i -e '$aGRUB_CMDLINE_XEN="dom0_mem=2048M,max:2048M"' /etc/default/grub
update-grub
sed -i -e 's/(enable-dom0-ballooning .*)/(enable-dom0-ballooning no)/' -e 's/(dom0-min-mem .*)/(dom0-min-mem 2048)/' /etc/xen/xend-config.sxp
reboot

Error: Dom0 dmesg log shows 'page allocation failure' or 'Out of memory: kill process:' or 'invoked oom-killer:' messages

Yes, these are vague symptoms, but I found that if I set vm.min_free_kbytes to a higher value this seemed to help. This may be partly precipitated by turning off dom0 ballooning and setting a fixed amount of dedicated memory. If you have lots of guests I think their I/O demands (disk and network) cause the dom0 kernel run out of wiggle room. Edit /etc/sysctl.conf and set the following option to reserve 128 MB for the kernel.

vm.min_free_kbytes = 131072

You can update this live with the following command.

sysctl vm.min_free_kbytes=131072

XENDOMAINS_SAVE

Edit /etc/default/xendomains and set XENDOMAINS_SAVE to be empty. This controls the feature that allows Xen to save the guest's running state when dom0 is shutdown. I almost never need this feature. It uses a lot of disk space.

#XENDOMAINS_SAVE=/var/lib/xen/save
XENDOMAINS_SAVE=""

xend won't start

I found that this happened when my dom0 ran out of disk space. For me the solution was, "don't run out of disk space".

guests start to have erratic networking

I found that this happened when my dom0 ran low on disk space. I am not certain that this is the cause because there were no useful messages in dmesg or any other log files .

Error: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.

If you see the following message in dmesg or /var/log/kern.log

Error: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.

then you probably need to patch /etc/xen/scripts/vif-common.sh and edit the function frob_iptables() so that it looks like the function below. You need to add the --physdev-is-bridged option to iptables in two places.

frob_iptable()
{
  if [ "$command" == "online" ]
  then
    local c="-I"
  else
    local c="-D"
  fi

  iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$vif" "$@" -j ACCEPT \
    2>/dev/null &&
  iptables "$c" FORWARD -m state --state RELATED,ESTABLISHED -m physdev \
    --physdev-is-bridged --physdev-out "$vif" -j ACCEPT 2>/dev/null

  if [ "$command" == "online" -a $? -ne 0 ]
  then
    log err "iptables setup failed. This may affect guest networking."
  fi
}