Increasing performance in KVM virtualization within a Tier-1 environment

Transcription

1 Journal of Physics: Conference Series Increasing performance in KVM virtualization within a Tier-1 environment To cite this article: Andrea Chierici and Davide Salomoni 2012 J. Phys.: Conf. Ser View the article online for updates and enhancements. Related content - A quantitative comparison between xen and kvm Andrea Chierici and Riccardo Veraldi - Integration of virtualized worker nodes in standard batch systems Volker Büge, Hermann Hessling, Yves Kemp et al. - Lxcloud: a prototype for an internal cloud in HEP. Experiences and lessons learned Sebastien Goasguen, Belmiro Moreira, Ewan Roche et al. Recent citations - Big Data Over a 100G Network at Fermilab Gabriele Garzoglio et al This content was downloaded from IP address on 12/02/2018 at 18:39

2 Increasing performance in KVM virtualization within a Tier-1 environment Andrea Chierici INFN-CNAF andrea.chierici@cnaf.infn.it Davide Salomoni INFN-CNAF davide.salomoni@cnaf.infn.it Abstract. This work shows the optimizations we have been investigating and implementing at the KVM (Kernel-based Virtual Machine) virtualization layer in the INFN Tier-1 at CNAF, based on more than a year of experience in running thousands of virtual machines in a production environment used by several international collaborations. These optimizations increase the adaptability of virtualization solutions to demanding applications like those run in our institute (High-Energy Physics). We will show performance differences among different filesystems (like ext3 vs ext4) when used as KVM host local storage. We will provide guidelines for solid state disks (SSD) adoption, for deployment of SR-IOV (Single Root I/O Virtualization) enabled hardware and what is the best solution to distribute and instantiate read-only virtual machine images. This work has been driven by the project called Worker Nodes on Demand Service (WNoDeS), a framework designed to offer local, grid or cloud-based access to computing and storage resources, preserving maximum compatibility with existing computing center policies and work-flows. 1. Introduction This work describes the optimizations we have been investigating and implementing at the KVM virtualization layer in the INFN Tier-1 at CNAF (Bologna, Italy), based on more than a year of experience in running thousands of virtual machines in a production environment used by several international collaborations. These optimizations increase the adaptability of virtualization solutions to demanding applications like those run in our institute (mostly related to High-Energy Physics). This work has been driven by the project called Worker Nodes on Demand Service (WNoDeS) [1][2], a framework designed to offer local, grid or cloud-based access to computing and storage resources, preserving maximum compatibility with existing computing center policies and work-flows. Published under licence by IOP Publishing Ltd 1

3 2. Testbed description 2.1. Hardware configuration The hardware we used for our testbed consists of two identical machines with the following characteristics: dual Intel Xeon GB of DDR3 2 Western Digital disks 300GB 10K RPM 16MB cache 2.5 SATA 3.0Gbps Internal Enterprise Hard Drive 82574L 1Gbps LAN adapter by Intel In order to perform some tests we added to this configuration: 160GB SATA II MLC Internal Solid State Drive 82599eb 10Gbps LAN adapter both provided by Intel. With recent CPU improvements it s also necessary to fine tune BIOS settings. In our case we used these options: hyperthreading disabled (no SMT) disks controller configured as AHCI raid disabled vt-d enabled virtualization support enabled SR-IOV enabled 2.2. Software configuration Our hypervisor run on a scientific linux 6.1 while the virtual machines run on scientific linux 5.7 ( SL from now on). Here is the detailed description of core packages installed: Software running on SL6 machine: kernel el6 glibc qemu-kvm libvirt Software running on SL5 machine: kernel el5 glibc kvm libvirt WNoDeS is currently used to virtualize EMI [3] Worker Nodes, so the operating system of the VM we tested is uniquely SL5. In the future, when EMI WNs are available for SL6, we will update our test environment Test software We run basically two sets of benchmarks, one for disk I/O and one for network I/O. 2

4 Disk We used iozone for disk measures with this specific command line: iozone -Mce -I -+r -r 256k -s <2xram>g -f /tmp/iozone -i0 -i1 -i2 Detailed description of this specific command line: -M iozone will call uname() and will put the string in the output file; -ce include flush (fsync,fflush) and close() in the timing calculations; -I use DIRECT I/O for all file operations; tells the filesystem that all operations are to bypass the buffer cache and go directly to disk; -+r enable O RSYNC and O SYNC for all I/O testing; -r # used to specify the record size, in Kb, to test; -s # used to specify the size, in Kb, of the file to test; -f filename used to specify the filename for the temporary file under test; -i # used to specify which tests to run. (0=write/rewrite, 1=read/re-read, 2=randomread/write) This test measures 6 core aspects of a disk performance: Read: Indicates the performance of reading a file that already exists in the filesystem; Write: Indicates the performance of writing a new file to the filesystem; Re-read: After reading a file, this indicates the performance of reading a file again; Re-write: Indicates the performance of writing to an existing file; Random Read: Indicates the performance of reading a file by reading random information from the file; i.e this is not a sequential read; Random Write: Indicates the performance of writing to a file in various random locations; i.e this is not a sequential write; Network We used netperf to test network throughput, with this specific command line: netperf -t TCP STREAM -H <dest-host> -l 180 Following is a detailed description of this specific command line: -t TCP STREAM indicates desire to measure TCP protocol performance; -H <dest-hosts> indicates the destination host of the stream to measure; -l 180 indicates the time in seconds for the test; We used lmbench to test network latency, with this specific command line: lat tcp -N 5 <dest-host> lat udp -N 5 <dest-host> -N 5 parameter indicates that test will be repeated 5 times in a row, to make a more accurate measure. We run tests for both TCP and UDP protocols. In both netperf and lmbench tests, the second host in our testbed was used as a server, to receive (and for inbound tests, transmit) the network streams. All the rpm packages containing the software were taken from the DAG[4] repository. 3

5 3. Disk I/O Disk I/O is a major issue in virtualization technologies. WNoDes currently uses KVM-based VMs, exploiting the KVM -snapshot flag. This allows to download (either via http or posix I/O) a single read-only VM image file to each hypervisor, and run VMs writing automatically purged delta files only. This saves substantial disk space and time, which would be needed if one had to locally replicate multiple images. Performance characteristics of this solution (disk caching does not allow us to publish a reliable benchmark) and the latest enhancements in qcow2 image handling pushed us to investigate the possibility to improve this solution What we tested We investigated performance differences among these solutions: raw image: default libvirt option, where the virtual disk is stored on an uncompressed image file, currently the most performing option among image files; ext4 file system: the default file system for SL6, declared to be faster and more reliable than the classic ext3. Since it s been back-ported to SL5, we wanted to test if using ext4 as file system for virtual image can improve performance; qcow2 standard image qcow2 image file without any particular optimization (no preallocation of metadata); qcow2 with metadata pre-allocation: qcow2 image file with some internal image setup to speed-up I/O performance (metadata pre-allocation); qcow2 image snapshot: allows to create a new image that refers to an original image (so called a backing file) using Redirect-on-Write [5]; any changes to the snapshot will not be reflected in the original image, but will be permanently stored in the snapshot image (also called delta image). The most interesting solution to investigate is qcow2 image snapshot with backing file. In this particular approach, a new external copy-on-write image is created, that can be used as a different image file for our virtual machines. Any modification made inside the virtual machine will only be reflected in this delta file image, leaving the original (also called backing file) image unchanged. If we test the write speed of the newly created image, according to the specification of a standard qcow2 image (a snapshot is exactly this), the image file has to grow as much as the size of the test file we use in iozone. This process of disk growth is time and resource consuming, giving as a result poor perfomance. If we run the test a second time, the disk is already big enough to contain the file iozone uses to test, and the general perfomance measured (particularly the write speed) is optimal. According to figure 1, if one excludes write performance, the remaining measures are rather similar. Writing performance is problematic and the difference between various solutions is more than 50% in some cases. The solution of qcow2 with metadata pre-allocation did not behave as expected for what concerns write speed and exhibited a significant performance degradation. As expected, a promising solution is the qcow2 image snapshot: indeed, in the second run of the test, write speed was the same as raw image, providing a significant boost if compared to other solutions investigated. The main problem, as we just explained, is the fact that we need to allow the disk snapshot to expand in order to reach good results. In figure 2, results similar to previous are illustrated running VMs on an SSD disk. With this kind of disk its easy to see the general performance boost among all solutions. In this case qcow2 with metadata pre-allocation behaves as expected, in some cases even better than raw image. The most interesting solution even in this test is the one with backing file on qcow2 image. Indeed we put the snapshot on the ssd but the backing file was still on the hdd as in previous test. The performance boost is really remarkable, and there is no need to allow the snapshot to expand to reach maximum write speed as in previous test. 4

6 Figure 1. Hard disk drive I/O performance From this bunch of tests we can claim that I/O virtualization has improved a lot with the latest KVM provided on SL6.1. Raw image is still appealing for general purpose but qcow2 format is more appealing for its enhanced features. Ext4 FS is clearly not mature enough on SL5 and so we discourage its adoption on production worker nodes. Current WNoDeS solution adopting snapshot option (that provides performance similar to raw image) could be substituted by qcow2 snapshot with backing file, particularly if adopting an ssd disk to store delta files (qcow2 snapshots). 4. SR-IOV The PCI-SIG (PCI Special Interest Group) developed the Single Root I/O Virtualization (SR- IOV) specification. The SR-IOV specification is a standard for a type of PCI passthrough which natively shares a single device to multiple guests. SR-IOV reduces hypervisor involvement by specifying virtualization compatible memory spaces, interrupts and DMA streams. SR-IOV improves device performance for virtualized guests. SR-IOV enables a Single Root Function (for example, a single Ethernet port), to appear as multiple, separate, physical devices. A physical device with SR-IOV capabilities can be configured to appear in the PCI configuration space as multiple functions, each device has its own configuration space complete with Base Address Registers (BARs) Advantages of SR-IOV SR-IOV [6] devices can share a single physical port with multiple virtualized guests. Virtual Functions have near-native performance and provide better performance than para-virtualized drivers and emulated access. Virtual Functions provide data protection between virtualized guests on the same physical server as the data is managed and controlled by the hardware and not by the software. These features allow for increased virtualized guest density on hosts within 5

7 Figure 2. Solid state drive I/O performance Figure 3. SR-IOV network card throughput 6

8 Figure 4. SR-IOV network card latency a data center. In other words, SR-IOV is able to better utilize the bandwidth of devices with multiple guests Enabling SR-IOV on a KVM host In order to enable SR-IOV support on a KVM hosts it s necessary to: Enable the Intel VT-d or AMD-Vi extensions in the BIOS of the host Activate I/O MMU in the kernel by appending intel iommu=on to the kernel line in the /boot/grub/grub.conf file; nothing is required for AMD hardware; Activate Virtual Functions within the network card kernel module: max vfs=<0..63> modprobe ixgbe 4.3. Tests performed We performed several tests to verify if SR-IOV enabled network cards are really performing as vendors claim [7]. Our test focused on a 10Gbps network card. We measured inbound and outbound connectivity, as well as latency. As it can be seen in figure 3 and 4, the 10Gbps SR- IOV NIC provides excellent performance, both for latency and aggregate throughput: indeed there is no significant difference in throughput between VMs and host, except for unexpected low performance when running a single VM instance, to be investigated further. Network latency is close to bare metal hardware, 3 times better than virtio. Virtio is a Linux standard for network and disk device drivers where just the guest s device driver knows it is running in a virtual environment, and cooperates with the hypervisor: this enables guests to get high performance network and disk operations. We think that SR-IOV tecnology is mature enough to be adopted in production environment, particularly with application where network latency is a major issue. 7

9 Figure 5. SCP-Tsunami vs SCP-Wave vs Plain Copy 5. Image distribution An important topic in the WNoDeS virtualization environment is the need to distribute the same image file to multiple virtualization hosts. Currently no specific tool is used, creating a potential bandwidth problem every time a new image has to be deployed on hypervisors. We tested several solutions and the most functional one is currently SCP-Tsunami. SCP-Tsunami [8] is a Python script that splits the file/images in chunks and transfers multiple chunks between virtualization hosts. With this simple script it s possible to pre-stage a VM image to a large set of nodes efficiently. SCP-Tsunami resembles the bittorent protocol but does not require the same complicate setup. SCP-Tsunami is a major improvement to SCP-Wave which offers a logarithmic speed-up, not enough for our production environment. In figure 5 one can see the remarkable performance boost compared to the current solution adopted by WNoDeS (plain image copy). Every node owning the image file contributes to spread the image to others, drastically reducing the time required to complete the copy operation. 6. KVM best practices Several documents are available on the web suggesting best practices for the optimization of KVM (see for example [9]). Here we highlight them according to our personal experience in particular for the virtualization of EMI Worker Nodes: Use KVM para-virtualized drivers for disk, memory and network: this is the starting point for every other optimization; Use if possible block devices for VM storage: we already investigated in previous work [10] that a guest operating system using block devices achieves lower-latency and higher throughput; Asynchronous I/O model for KVM guests: using AIO (aio=native) support can improve guest I/O performance, especially when there are multiple threads performing I/O operations at the same time; Disk caching: use the writeback option where both the host page cache and the disk write cache are enabled for the guest. 8

10 During our investigations we tested some of the suggested optimizations that did not meet the expectations in our tests. For example, some best practices suggest to overcommit memory and cpu. An EMI worker node in our computing farm is 99% of the time fully loaded, so overcommitting, particularly cpu, is a very bad idea, because of reducing performace significantly. Another hot topic in virtualization optimizations is I/O schedulers. We did some prelimiar tests and discovered that virtualizing EMI Worker Nodes, changing scheduler (sometimes called also elevator ) does not affect performance. Deadline and CFQ elevators make sense in a different environment, where fiber channel storage is attached to the node to be tested, clearly not our case. 7. Future work This work showed some interesting results, anyway we need to continue testing new solutions that are continuosly made available. The recently released scientific linux version 6.2, has been advertised as a significant improvement in virtualization performance, but we were not able to test it on time for this work. Moreover, as soon as SL6 EMI worker nodes will be available for general use, we will test performance differences compared to SL5 one, and we are sure there will be a general improvement: indeed, some solution for advanced resource optimization, like KSM [11] and huge pages[12], are available only starting from SL6 (backports have been done on SL5, but are just proof-of-concepts, not ready for production). 8. Conclusions In this work we tried to show if and how it s possible to increase performance in the virtualization solution currently adopted by WNoDeS project. We showed that qcow2 snapshot with backing file is a good solution for creating a new image, as compared to the current KVM snapshot approach. This solution can be a major improvement to easily manage updates of VMs on WNoDeS hypervisors: right now applying a kernel or security fix triggers a new copy of the whole image, while with the qcow2 snapshot one would simply have to copy the delta file (generally just a few megabytes, compared to some gigabytes of the whole image) leaving the original image (the backing file) copied previously untouched. We examined solid state drive perfomance and, as expected, a significant boost in perfomance is measurable in every aspect of the disk I/O. Since ssd disks are still rather expensive, we think that a good compromise could be to use such a disk for storing only the qcow2 snapshots of images archived on standard hdd drives. Network perfomance has not been a problem since the very beginning of the project and we proved that 10Gbps networks with SR-IOV technology is a significant improvement in a very high demanding environment or if the network throughput/latency required is very high. Finally we showed that using a tool like SCP-Tsunami, is extremely convenient in order to be able to rapidly distribute image files across several different virtualization hosts. Appendix A. qemu-img usage guide In this paper we talked about some special features of qcow2 images: in this section we will show the command lines required for that. Appendix A.1. metadata preallocation The first feature of qcow2 format we talked about is the metadata preallocation. Generally if we generate an image in qcow2 format (either using virt-manager or not) here is what we get on a standard SL6 hypervisor: # qemu-img create -f qcow2 sample.img 5G Formatting sample.img, fmt=qcow2 size= encryption=off cluster_size=0 9

11 # qemu-img info sample.img image: sample.img file format: qcow2 virtual size: 5.0G ( bytes) disk size: 136K cluster_size: # ll -h grep sample -rw-r--r-- 1 root root 256K May 22 15:36 sample.img As we can see the image is created with a disk size of 136K (ls shows 256K) even if the virtual size we required is 5Gb. Now let s see how it s possible to preallocate metadata information on the image and how this affects file size: # qemu-img create -f qcow2 sample.img 5G -o preallocation=metadata Formatting sample.img, fmt=qcow2 size= encryption=off cluster_size=0 preallocation= metadata # qemu-img info sample.img image: sample.img file format: qcow2 virtual size: 5.0G ( bytes) disk size: 912K cluster_size: # ll -h grep sample -rw-r--r-- 1 root root 5.1G May 22 15:35 sample.img Now it s clear something changed. The ls command shows a 5Gb file and qemu-img shows a disk size of 912K, bigger than in previous case. Appendix A.2. snapshot To create a snapshot the process is similar to the one we have seen in previous section and again we need to use the qemu-img command by hand: # qemu-img create -f qcow2 -b sample.img snapshot.img Formatting snapshot.img, fmt=qcow2 size= backing_file= sample.img encryption=off cluster_size=0 # qemu-img info snapshot.img image: snapshot.img file format: qcow2 virtual size: 5.0G ( bytes) disk size: 136K cluster_size: backing file: sample.img (actual path: sample.img) The info about the snapshot are clearly displayed and the image that we took the snapshot of is indicated as the backing file. Please remember that no changes have to occur to the backing file once the snapshot is taken or unpredictable behavior will happen. Currently these advanced image manipulation features are not supported under libvirt. 10

12 Appendix B. 10Gbps Optimizations In order to achieve high throughput with 10Gbps network cards, it s necessary to perform some tuning on a standard SL distribution. Here is what we used in our tests (see also [13]). #!/bin/bash echo "Optimise TCP parameters" # enable timestamps, window scaling echo 1 > /proc/sys/net/ipv4/tcp_timestamps echo 1 > /proc/sys/net/ipv4/tcp_window_scaling echo 1 > /proc/sys/net/ipv4/tcp_moderate_rcvbuf echo > /proc/sys/net/core/wmem_max echo > /proc/sys/net/core/rmem_max echo > /proc/sys/net/core/rmem_default echo > /proc/sys/net/core/wmem_default echo " " > /proc/sys/net/ipv4/tcp_rmem echo " " > /proc/sys/net/ipv4/tcp_wmem # echo "Optimise ethernet queue" /sbin/ifconfig eth0 txqueuelen # Jumbo Frames /sbin/ifconfig eth0 mtu 9000 # enable path mtu discovery sysctl -w net.ipv4.tcp_mtu_probing=1 sysctl -w net.ipv4.ip_no_pmtu_disc=0 # re-enable SACK sysctl -w net.ipv4.tcp_sack=1 # increase backlog ("rxqueuelen"): sysctl -w net.core.netdev_max_backlog= # discard metrics of old connections: sysctl -w net.ipv4.tcp_no_metrics_save=1 # select htcp congestion control algorithm: # (or, another interesting one, bic) sysctl -w net.ipv4.tcp_congestion_control=htcp References [1] Salomoni D et al 2011 J. Phys.: Conf. Ser. 331 [2] WNoDeS Website: [3] EMI Website: [4] DAG Repository: [5] [6] root/ [7] [8] SCP-Tsunami webpage: 11

13 [9] Virtualization Best Practices: pdf.pdf [10] Chierici A, Salomoni D, Veraldi R 2009 Measuring performances of linux hypervisors Il Nuovo Cimento Vol.32 C, N. 2 [11] [12] Hat Enterprise Linux/6/html/Performance Tuning Guide/smemory-transhuge.html [13] Chierici et al Performance of 10 Gigabit ethernet using commodity hardware IEEE Transactions on Nuclear Science Vol.57, N. 2 12