Migrace z Xenu na KVM

Migration of VMs between Xen and KVM - back and forth - the simple way

I am writing this in English because I spent one whole day by googling and I found no info about this even being possible. So I spent another day by frustrating experiments with no results, only to find very simple way which made me happy.

What we want to achieve

Migration of VMs between two Xen and KVM machines.

The migration does not have to be online, but it must be quick and easy.

The Xen is Debian 5 Lenny, KVM is Ubuntu 10.04 (~Debian Squeeze).

The XEN VM guests were created by the xen-create-image tool and we have too many of them, some with Debian 4, some with Debian 5, some with Debian 6. If all of these should be reinstalled from scratch, that would take month of work!

The migration is possible because XEN is great and flexible system, allowing to customize all VMs in a way that they can be booted in both XEN and KVM with no intermediate modifications necessary.

I do not guarantee that this will work in XEN 4 (Debian Squeeze XEN host), because I can't try it currently.

Why I need this

Because:

• I need XEN for its easy PCI and USB passthrough on older HW without the VT-d support (KVM can't do PCI or USB passtrhrough on anything older than the i5500 chipset).
• I've been using XEN for more than 5 years and I love it. It is rock-stable, it never failed. As for KVM, we have started using it year ago, it seems to work, but this is all I can say about it now.

How to do it

First of all, setup and start the DRBD devices, which will replicate your VM disks from the XEN to the KVM physical host machines.

Then, you must customize the XEN VM guests the following way:

/etc/inittab in all VM guest

You must comment out the following line, which is created by xen-create-image and directs console to the hvc0, which does not work in KVM:

#1:2345:respawn:/sbin/getty 38400 hvc0

Instead, place the standard tty1 console definition there:

1:2345:respawn:/sbin/getty 38400 tty1

Now, the console will not work in XEN. For enabling XEN console, you can place another line:

co:23:respawn:/sbin/getty -L hvc0 9600 vt102 

Or you can redirect the hvc0 console to the tty1 in the VM guest definition file by placing addition line

extra = "console=hvc0 xencons=tty"

Modify VM guest definition files

You must rename your disks to sda/sdb/sdc... in the guest definition files (/etc/xen/auto/*.cfg).

By default, xen-create-image generates disk names with partition numbers in the form (hd|sd|xvd)[a-z][0-9], e.g. hda1, sda1 or xvda1

The problem is, that

• KVM can only handle disk names without partition numbers:
  • vd[a-z] when you use the virtio type
  • sda[a-z] when you use the "IDE" type
  • in next step, while still running the VM guests in XEN, we will install kernel and grub in each VM guest, but grub/grub2 in Lenny and Squeeze (grub2 versions < 1.99) have a bug #601974 which does not allow installing it into a partition which is not part of a whole disk.

Therefore the only possible way forward is to use the sda/sdb/sdc... disk names in XEN.

Fortunatelly, XEN is happy with sda/sdb/sdc disk names so you can modify the guest definition files to like this:

#
# Configuration file for the Xen instance logger, created
# by xen-tools 3.9 on Wed May 19 02:56:03 2010.
#
#
#  Kernel + memory size
#
kernel      = '/boot/vmlinuz-2.6.26-2-xen-amd64'
ramdisk     = '/boot/initrd.img-2.6.26-2-xen-amd64'
memory      = '2048'

#
#  Disk device(s).
#
root        = '/dev/sda ro'
disk        = [
                  'phy:/dev/vg_main/logger-swap,sdc,w',
                  'phy:/dev/drbd5,sda,w',
                  'phy:/dev/drbd6,sdb,w',
              ]
cpus="0-7"
vcpus=8
extra="clocksource=jiffies"
#for standard /etc/inittab (compatible with KVM):
extra = "console=hvc0 xencons=tty"
#
#  Hostname
#
name        = 'logger'
#
#  Networking
#
dhcp        = 'dhcp'
vif         = [ 'mac=00:16:3E:37:D0:70' ]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Change /etc/fstab in each guest VM

Before you restart (re-create) the guest VM, log into it and modify its /etc/fstab mount points to comply with your new disk names !

Install grub and kernel in each VM guest

Install Grub - it should be happy now with your /dev/sda root disk:

apt-get install grub2

Install the proper kernel file - e.g., for Debian Squeeze VM guest:

apt-get install linux-image-2.6.32-5-amd64

Re-create and test the guest VM in XEN

Either

poweroff

in guest or

xm shutdown logger

in host, and then

xm create -c /etc/xen/auto/logger.cfg

Create new guest defintion file for KVM

On the KVM machine, thanks to DRBD, you already have the guest partitions ready, so the last step is to create the proper guest definitions, e.g. /root/logger.xml :

<domain type='kvm'>
 <name>logger</name>
 <uuid>30ea508e-7ed4-d590-3f63-0ea9d22e2545</uuid>
 <memory>2097152</memory>
 <currentMemory>2097152</currentMemory>
 <vcpu>2</vcpu>
 <os>
   <type arch='x86_64' machine='pc-0.12'>hvm</type>
   <boot dev='hd'/>
 </os>
 <features>
   <acpi/>
   <apic/>
   <pae/>
 </features>
 <clock offset='utc'/>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>restart</on_crash>
 <devices>
   <emulator>/usr/bin/kvm</emulator>
   <disk type='block' device='disk'>
     <driver name='qemu' type='raw'/>
     <source dev='/dev/drbd5'/>
     <target dev='sda' bus='ide'/>
   </disk>
   <disk type='block' device='disk'>
     <driver name='qemu' type='raw'/>
     <source dev='/dev/drbd6'/>
     <target dev='sdb' bus='ide'/>
   </disk>
   <disk type='block' device='disk'>
     <driver name='qemu' type='raw'/>
     <source dev='/dev/vlam-server/logger-swap'/>
     <target dev='sdc' bus='ide'/>
   </disk>
   <interface type='bridge'>
       <mac address='00:16:3E:37:D0:70'/>
     <source bridge='br0'/>
     <model type='virtio'/>
   </interface>
   <console type='pty'>
     <target port='0'/>
   </console>
   <console type='pty'>
     <target port='0'/>
   </console>
   <input type='tablet' bus='usb'/>
   <input type='mouse' bus='ps2'/>
   <graphics type='vnc' port='-1' autoport='yes'/>
   <video>
     <model type='cirrus' vram='9216' heads='1'/>
   </video>
 </devices>
</domain>

You can test the definition syntax by

virsh define /root/logger.xml

Testing the guests in KVM

Before you start your guest in KVM, I very recommend to make "dry run" test on image snapshot:

lvcreate /dev/vlam-server/logger-system --snapshot -n logger-system-snap -L 2G
lvcreate /dev/vlam-server/logger-var --snapshot -n logger-var-snap -L 2G

(Note: /dev/vlam-server/logger-* are the LVM partitions underlying the /dev/drbd5 and drbd6 devices).

Now, modify the KVM guest definition in order to use the snapshot instead of DRBD devices and start the VM by

virsh create /root/logger.xml

Switch KVM's DRBD to primary and run

If everything works OK, then modify the KVM definition back to /dev/drbd* devices and:

On XEN host, shutdown the VM:

xm shutdown logger

On XEN host, switch your DRBD devices to "secondary":

drbdadm secondary logger-system logger-var

On KVM host, switch your DRBD devices to "primary":

drbdadm primary logger-system logger-var

On KVM host, start the VM:

virsh create /root/logger.xml

Klímova crazy migrace z KVM na XEN

• Pomocí příkazu scp si překopírujeme obraz odpojeného virtuálu ze serveru s Xenem do server s KVM (pokud vám neběží jako nám na stejné mašině).

scp root@kraken.slfree.czf:/store/xen/domains/vpn/disk.img /store/disk.img

• Nyní si na vytvoříme na KVM nový virtuál, kterému přiřadíme nový prázdný VIRTIO disk o velikosti malinko větší než je velikost zkopírovaného obrazu (+ ještě něco pro swap). Dále mu pak přidáme IDE disk, což bude samotný zkopírovaný obraz. Nakonec je ještě potřeba tomuto virtuálu nastavit bootování z CD-ROM, kterému jako obraz dáme některou LiveCD distribuci (já použil Ubuntu 10.10). Je vhodné si nastavit síť pro přístup do Internetu.
• Po nabootování virtuálu do LiveCD spustíme terminál a přihlasíme se jako root (v Ubuntu otevřít gnome-terminal a napsat sudo -s).
• Pomocí programů gparted nebo fdisk na VIRTIO disku vytvoříme novou tabulku oddílu a poté vytvoříme 2 oddíly - jeden malý pro boot (já dal 236MB ext2, nezapomenout nastavit příznak pro bootování!) a druhý nezformátovaný pro LVM.
• Doinstalujeme lvm, vytvoříme výchozí VG a LV pro systém a swap.

apt-get install lvm2
pvcreate /dev/vda2
vgcreate rootg /dev/vda2
lvcreate -L velikost pro root -n root rootvg
lvcreate -L velikost pro swap -n swap rootvg

• Vytvoříme swap.

mkswap -f /dev/mapper/rootvg-swap

• Překopírujeme obraz původní virtuálu:

dd if=/dev/sda of=/dev/mapper/rootvg-root

• Zvětšíme souborový systém do konce oddílu:

resize2fs /dev/mapper/rootvg-root

• Připojíme si oddíly a do systému se chrootnume:

mount /dev/mapper/rootvg-root /mnt
mount /dev/vda1 /mnt/boot
mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount -o bind /dev /mnt/dev
chroot /mnt /bin/bash

• Doinstalujeme lvm, jádro a grub2.

apt-get install lvm2 linux-image-2.6.26-2-amd64 grub2

• Pouze pro Debian Lenny: Právě tady nastává u lennyho problém. Možná se vám stalo, že se vám jádro nepodařilo zkonfigurovat. Na vině je bug v dmsetupu. Opravíme ho následujícím způsobem:

rm /dev/mapper/rootvg-root
cp -R /dev/dm-0 /dev/mapper/rootvg-root