XEN and KVM in INFN production systems and a comparison between them Riccardo Veraldi Andrea Chierici INFN - CNAF HEPiX Spring 2009
Outline xen kvm Test description Benchmarks Conclusions Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 2
What we have now: Xen Open source industry standard for virtualization Included in all recent linux distributions Supports para and hvm approaches Widely used in HEP community Won t go into details during presentation Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 3
What could be the future: KVM Kernel-based Virtual Machine Open source included in latest linux kernels: implemented as a module A user space program uses /dev/kvm interface to set up VMs (qemu-kvm) Supports hvm approach Rather new to HEP community Qumranet now owned by Red Hat Foreseen boost in development Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 4
KVM virtio Direct access to hardware Hard disk (/dev/vdx) Ethernet interface Support from kernel 2.6.25 Supported since RHEL 5.3 (virtio backport) kernel 2.6.18-128.1.1 Virtio drivers must be on the VM, not on the host EGEE sw can t currently benefit from this Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 5
KVM seen by system admin rpms: basically the kernel module and a modified version of qemu (qemu-kvm) Network configuration to be done by hand in order to get public IP bridge-utils and tunctl rpms were used to set-up tap interface Script developed to automate the process A modified init.d script to configure software bridge has been developed at CNAF No VM configuration file direct support VMs are launched via a standard UNIX command + command line options libvirt Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 6
KVM seen by system admin No direct interface to manage VMs by command line (e.g. xm ) Libvirt support Virt-manager working Monitor interface Very powerful, lots of options available qemu already well documented, support available on-line Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 7
KVM introduced at CNAF We use quattor to configure and install grid nodes and virtual machines Profiles describe machine configuration Network boot for installation working Keeps booting from network, need to restart with -boot c No need for any modification, just like xenhvm Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 8
Qualitative test CNAF is running LHCb tier2 site entirely on xen VMs (2 CEs, 1 SE) Changed one Computing Element with a KVM machine Has been working for more that 3 weeks flawlessly Quattor machine profile unmodified, no effort for sysadmin CMS secondary squid server installed on the same host with same result (2 weeks ago) Running a KVM in production since 2 months for INFN-CA backups. Works flawlessly KVM executed (and live migrated!) a win7 VM Hardware used: 1 node, dual E5420, 16GB ram, sata disks via Areca controller Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 9
Quantitative test: description Need some measures to understand what is the best solution Tested 3 classic parameters CPU hep-spec06 (v1.1), PI test for cpu distribution Network iperf (v2.0.4) Disk access bonnie++ (v1.94) Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 10
Quantitative test: description Hardware used: 1 blade, dual E5420, 16GB ram, 10k sas disk via LSI logic raid controller (raid0) Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a file Xen-hvm VM specs: 1 vcpu, 2GB ram, disk on a file, netfront network driver KVM VM specs: 1 vcpu, 2GB ram, disk on a file, e1000 network driver emulation (non-virtio tests) Host OS: SL 5.2 x86_64, kernel 2.6.18-92.1.22.el5 VM OS: SLC 4.5 i386, kernel 2.6.9-67.0.15.EL.cern VM OS: SLC 5.3 x86_64, kernel 2.6.18-128.1.1 for kvm virtio tests KVM version: 83 Xen version: 3.2.1 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 11
Benchmarks: HEP-Spec06 11 XEN vs. KVM on dual Intel E5420, single performance measure 10.5 10 HEP Spec06 9.5 9 8.5 8 7.5 1 4 7 8 Number of concurrent VMs Xen hvm Xen para KVM E5420 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 12
Benchmarks: HEP-Spec06 VMs vs. CPU 70 60 50 HEP Spec06 40 30 20 10 0 1vm 4vm 7vm 8vm kvm xen hvm xen para E5420 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 13
Benchmarks: HEP-Spec06 8VMs aggregate vs. CPUs 70 60 50 HEP Spec06 40 30 20 10 0 E5410 E5420 E5420kvm E5420xen hvm E5420xen para E5430 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 14
Benchmarks: HEP-Spec06 Virtualization Technology % loss from non emulated CPU (E5420, 8vm) E5420kvm 3,42 E5420xen-hvm 4,55 E5420xen-para 2,02 E5410 vs. E5420 4,07 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 15
Benchmarks: PI test System tools to measure CPU usage are not reliable enough Calculating PI with n arbitrary ciphers Algorithm based on Numerical Recipes Written in C Compiled with options gcc -O3 -ffast-math -finline-limit=1000 Returns the number of CPU cycles An external function included into the code calculates the number of CPU cycles, using rdtsc IA32 registry Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 16
Benchmarks: PI test KVM host vs. guest 3.1E+12 3E+12 2.9E+12 CPU cycles 2.8E+12 2.7E+12 2.6E+12 2.5E+12 2.4E+12 2.3E+12 2.2E+12 2.1E+12 difference < 2% host guest Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 17
Benchmarks: PI test 8 concurrent guest CPU cycles 3.1E+12 3E+12 2.9E+12 2.8E+12 2.7E+12 2.6E+12 2.5E+12 2.4E+12 2.3E+12 2.2E+12 2.1E+12 guest singolo guest1 guest2 guest3 guest4 guest5 guest6 guest7 guest8 8 core Balanced Performance Every guest uses one core at 100% Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 18
Benchmarks: PI test 17 concurrent guests 7E+12 6E+12 5E+12 CPU cycles 4E+12 3E+12 2E+12 1E+12 Good balancing on guest Max difference 14%, average ~5% Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 19
Benchmarks: Iperf KVM Network Performance, e1000 1000 900 800 700 Mbits/sec 600 500 400 300 200 100 in out 0 1vm 2vm 4vm 8vm host iperf w256k P 5 t 900 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 20
Benchmarks: Iperf KVM vs XEN, 1vm, network performance 1000 900 800 700 Mbits/sec 600 500 400 300 200 100 in out 0 kvm e1000 kvm virtio xen para xen hvm pv iperf w256k P5 t 900 Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 21
Benchmarks: bonnie++, file vs lvm 2GB Ram, 4GB data set, 1vm comparison 160000 140000 120000 100000 K/sec 80000 60000 40000 20000 0 Host 1 kvm file 1 kvm lvm 1 xen para file 1 xen para lvm 1 xen hvm file 1 xen hvm lvm bonnie++ on a single VM seq out block seq out rewrite seq in block Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 22
Benchmarks: bonnie++ 2GB ram, 4GB data set, single 120000 100000 80000 60000 40000 20000 0 1 kvm vm 4 kvm vm 8 kvm vm 1 xen para 4 xen para 8 xen para 1 xen hvm 4 xen hvm 8 xen hvm seq out block seq out rewrite seq in block Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 23
Benchmarks: bonnie++, kvm virtio 32GB data set, lvm partition, single 140000 120000 100000 K/sec 80000 60000 40000 20000 0 Host hdb vdb Bonnie++ on a single vm seq out block seq out rewrite seq in block Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 24
Benchmarks: bonnie++ 2GB ram, 4GB data set, 8vm aggregate 120000 100000 80000 K/sec 60000 40000 20000 0 seq output block seq output rewrite seq input block Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 25
Conclusions KVM proved good stability and reliability No problems on running production machines for more than 4 weeks CPU performances are extremely good Modern CPU virtualization technologies help! Network performances are fair with e1000, good with virtio Disk I/O seems the most problematic aspect Other solutions have problems too Requires sysadmins only a small effort Even if looking promising, right now xen is the most performing solution Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 26
Future work qemu snapshot features Backup and cloning of nodes high-level VM managers Ovirt, enomalism, ganeti Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 27
Bibliography and links Xen Xen repository KVM Kvm repository Quattor HEP-SPEC Numerical Recipes: second edition in C Computer Systems: A Programmer's Perspective Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 28