Performance evaluation of private cloud computing with Eucalyptus

SCIS & ISIS 2010, Dec. 8-12, 2010, Okayama Convention Center, Okayama, Japan Performance evaluation of private cloud computing with Eucalyptus Kei Hirata 1, Akihiro Yamashita 1, Takayuki Tanaka 2, Masaya Arai 1, Ryota Ono 1, Hidenori Kawamura 1, Keiji Suzuki 1 1 Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Hokkaido, Japan 2 Creationline Inc., Minatoku, Tokyo, Japan AbstractCloud computer technologies are advanced better, and web service like IaaS is used as businesses for its easiness and flexibility. But people concern about security on the cloud service and still feel resident to storing enterprise data in external server. So people interest in private cloud environment. In this paper, we evaluate private cloud system including Eucalyptus, an opensource software implementation of cloud computing system like Amazon EC2. Evaluations are about basic performance, management and recovery, web application performance for this system. Finally, we conclude if limited to small-scale, enterprise private infrastructure for short time use, it can be a good private cloud system. I. INTRODUCTION As the virtualization technologies advance, networked service, such as SaaS (Software as a Service) and PaaS (Platform as a Service) are becoming more and more popular. The demands and interests of cloud computing services are rapidly increasing. Among them, Amazon EC2[1], one of the IaaS (Infrastructure as a Service), is used as business platform by small and large bussinesses for its easy installation and clear pricing policy. Using cloud computing technologies, hardware is resourced to reduce the maintenance cost of the system and to quickly set up a new system. But, people still feel resistant to storing the enterprise data in external cloud environment. These circumstances lead to more and more interests in private cloud environment for the enterprises, in particular the cloud environment implemented with open source software (OSS) for its low cost and customizability. However, OSS cloud environment is a new field with few cases and limited documentation. In this paper, we evaluate the cloud management system. The system includes Eucalyptus[2], an OSS cloud management tool and compatible with Amazon EC2/S3 EBS at API level, with Zabbix[3], an integrated monitoring tool of physical and virtual machines and other telecommunication devices, and virt-manager[4], a virtualization management tool. At this survey, we focus on the feasibility of the whole system as an enterprise cloud environment, rather than a detailed evaluation of each tool's functionality. II. SUMMARY OF EVALUATION 1) Introduction to evaluation The purpose of the evaluation is to supply the information required for the cloud management system with Eucalyptus. We evaluated basic performance evaluation, management and recovery Evaluation, web application performance evaluation for this system. 2) Eucalyptus Eucalyptus (Elastic Utility Computing Architecture for Linking YourPrograms To Useful System) is an open-source software implementation of cloud computing that utilize compute resources[5]. Eucalyptus provides compute resource and infrastructure as VM (Virtual Machine) instance and Virtual Network. User can create VM instances that available some characteristic option about computer performance (CPU core number, memory capacity and disk size) from compute resource, locate network address for VM instances, and use block storage service like Amazon S3/EBS service for adding disk and backup. Administrator can configure network security policy and filter network packets in cloud system. In essence, the system allows users to start, control, access, and terminate entire virtual machines using an emulation of Amazon EC2's SOAP and "Query" interfaces. Eucalyptus supports Xen[6] and KVM/QEMU[7] for virtualization. Eucalyptus system has three-layered hierarchical structure, and each layer has some high-level components. Each component has own Web-service interface and role as below for implement cloud service. Cloud Controller (CLC) is the entry-point into the cloud for users and administrators. It queries node managers for information about resources, makes highlevel scheduling deecisions, and implements them by making requests to cluster controllers. Walrus is Amazon S3 API compatible storage server. Eucalyptus system store VM image or snapshot of the Block storage attached to VM instances. Walrus is mostly installed on the machine same as Cloud Controller. Cluster Controller (CC) gathers information about and schedules VM execution on specific node controllers, as well as manages virtual instance network. Storage Controller manages block storage service for VM instances. This component creates new block storage from local disk or restores snapshot from Walrus for VM instances. Storage Controller must be installed with Cluster Controller. 1015

Node Controller (NC) controls the exection, inspection, and terminating of VM instances on the host where it runs. The relationships of each component are shown in Figure 1. The tool used in the evaluation to measure performance is shown in Table 2. TABLE 2 SOFTWARE VERSION OF MEASURE TOOL Measure tool Version sysstat iperf bonnie++ 9.1.1 (build) 2.0.4 (build) 1.0.3e (build) Figure 1. relationships of components in Eucalyptus system 3) Evaluation environment The system structure of evaluation is shown in Figure 2. III. EVALUATION OF PERFORMANCE In this section, we describe basic performance evaluation of Eucalyptus as cloud management tool. List of evaluation test is shown below. Performance of instance start-up and stop Performance of network Performance of Block Storage 1) Performance of starting and stopping instances Figure 2. The system structure in the evaluation Eucalyptus system has two CCs and six NCs. Zabbix Server monitors physical machines and VM instances including Eucalyptus system. External Server are used in network traffic test or migration test. Each machine's model number is "HP ProLiant ML 110 G5" embedded CPU "Intel Xeon E3110" (3.0GHz 2Core), 4GB memory, and NIC "Broadcom BCM5722". In addition, CC embedded another NIC "HP NC110T" for connection to the Network Switch. In this evaluation, we chose KVM as it is integrated in Linux kernel and thus is expected becomemore popular in the future. The host OS is Fedora 12. The version of the software used in the evaluation is shown in Table 1. For testing the performance of starting and stopping instances, we prepared two VM images with different sizes and measured the time of starting and stopping one instance and multiple instances concurrently. Smaller VM image sizes 1GB, and CentOS 5.3(64bit) is installed in. This image is Eucalyptus System's official image. Another VM image sizes is 5GB, and installed Fedora 12. As Eucalyptus caches VM image in NC, we measured the time with and without cache TABLE 3 SETTING OF INSTANCE TYPES Name CPUs Memory (MB) Disk (GB) m1.small 1 128 6 m1.large 2 512 10 m1.xlarge 2 1024 20 TABLE 1 SOFTWARE VERSION USED IN THE EVALUATION ENVIRONMENT Software Linux kernel Eucalyptus euca2ools Sun JDK qemu-kvm libvirt vconfig aoetools vblade Version 2.6.32.9-70.fc12.x86_64 1.6.2 (build) 1.2 (build) 1.6.0_18 2:0.11.0-13.fc12 0.7.1-15.fc12 1.9-8.fc12 23-3.fc12 14-6.fc12 Figure 3. the relationship of instance type and the number of concurrently started instances and start time 1016

image. The instance types used to survey the performance of start and stop are listed in Table 3. The disk size of m1.small is, considering the VM image size, set to 6GB. The measurement results of start and stop time are in Figure 3. The results show that concurrent instance start may take a few minute delay. The instance start time depends on VM size, ranging from a few minutes to ten minutes; and stop time is approximately 10 seconds, and thus we conclude it is practical when fast start is not necessary. Note that there is a report that NC may fail to start in a high load situation, and we also had such cases when when started four instances concurrently. This bug is fixed in Fedora 13. Figure 5. Performance of four instance concurrent communication Eucalyptus uses e1000 emulation by default because it is less environment dependent. The e1000 emulation shows approximately 20 per cent of the physical performance while virtio shows approximately 80 per cent. Therefore, when the environment allows, it is recommeded to use virtio. It also performed well when concurrently communicated. We conclude that the network performance is sufficient. 3) Block Storage performance Block Storage is a function equivalent to AmazonEC2's Figure 4. the system structure of network performance evaluation 2) Performance of network KVM assigns a virtual NIC when an instance is started. In this evaluation, we configured virtio, a paravirtualization of e1000 emulation, and used iperf to measure the network performance. The system structure of network performance evaluation is in Figure 4. The measured bandwidths between an instance and a physical machine or between two instances are illustrated in Table 4, and the results of measured bandwidths of concurrent communication is illustrated in Figure 5. Figure 6. The results of Block Storage performance and comparison with local disk performance EBS. Eucalyptus implements it using AoE (ATA over Ethernet). The measured results of Block Storage performance is shown in Figure 6. Next, we evaluated the performance of Block Storage accessed concurrently by multiple instances. Figure 7 shows the case of four instances. Note that we measured three times to evaluate in the graph. The evaluation revealed that, when multiple users access to Block Storage, its I/O performance is distributed and becomes unstable. Eucalyptus uses vblade to implement AoE. We also measured the performance of AoE using vblade in a physical TABLE 4 THE COMMUNICATION PERFORMANCE BETWEEN AN INSTANCE AND A PHYSICAL MACHINE OR BETWEEN TWO INSTANCES e1000 target 1 direction target 2 bandwidth target 1 direction target 2 bandwidth VM01 292Mbps 2.61 Gbps VM03 VM03 224Mbps 2.28 Gbps VM02 571Mbps VM02 646 Mbps 545Mbps 633 Mbps 259Mbps 661 Mbps NC01 VM01 NC01 692Mbps 544 Mbps CC01 206Mbps CC01 904Mbps 765Mbps 606Mbps CLC 225Mbps CLC 902Mbps 708Mbps 609Mbps virtio 1017

TABLE 6 LIST OF SURVEY CHECK ITEMS check item summary Figure 7. The performance of four instance concurrent access to Block Storage environment, which gave us similar results. The implementation of vblade has no consideration of multiple users and does not have the function to control traffic. Therefore, when multiple instances concurrently access to vblade, it falls in poor performance or, in a worst case, becomes unaccessible. As conclusion, using Block Storage with multiple users should be avoided, and for data persistency, it is recommended that a separated file server is networked via NFS. Empty resource status Instance status Instance IP status EBS status SSH connection EBS recognition from EBS writable New EBS attach SSH logout SSH login again The content of empty resource status displayed in eucadescribe-availavility-zones The instance status displayed in euca-describe-instances The IP status of instance displayed in euca-describeinstances EBS status displayed in eucadescribe-volumes Checked the prompt of logged in connection Checks if the EBS device file attached exists Checks if EBS volume mounted is writable Checks if able to attach a new EBS Checks if able to log out SSH SSH login again IV. EVALUATION OF MANAGEMENT AND RECOVERY For management and recovery, we evaluated as follows: Failure handling of the cloud Migration of VM images 1) Evaluation of failure handling on Cloud Computing In this evaluation, we prepare 12 failure scenarios and 10 check list items. Table 5 shows the scenarios and Table 6 shows the check list. TABLE 5 LIST OF FAILURE SCENARIOS FOR FAILURE HANDLINF SURVEY availability. Failure handling is one of the important issues for Eucalyptus as cloud computing. 1) Migration of VM images For VM image migration, we evaluated as follows: Portability evaluation of VM images for KVM and VM images for Eucalyptus Migration from physical machine to Eucalyptus Portability evaluation from image on Amazon EC2 (AMI) to image on Eucalyptus (EMI) Fairule scenario CLC service down CLC service reboot CC service down CC service reboot NC service down NC service reboot CLC-router network disconnect CLC-router network reconnect CC-Hub network disconnect CC-Hub network reconnect CLC machine reboot + service boot CC machine reboot + service boot Summary Force quit CLC service Reboot CLC service from Force quit CC service Reboot CC service from Force quit NC service Reboot NC service from Disconnect the network between CLC and router Disconnect and reconnect the network between CLC and router Disconnect the network between CC and Hub Disconnect and reconnect the network between CC and Hub Reboot CLC machine and boot CLC service Reboot CC machine and boot CC service The evaluation uses KVM as the hypervisor of Eucalyptus, requiring the condition that the kernel image is able to work with KVM. As a result, VM images on KVM can work on Eucalyptus, thus, when a new VM image is registered for Eucalyptus, it is suggested that the kernel image works on KVM. The migration from physical machines (P2V) can be done by partimage, a partition management tool, copying the partition from physical machine to VM image. The VM images on Amazon EC2 (AMI) and on Eucalyptus (EMI) are compatible and can be transferred from one to the other. The operation interface of EC2 has been changed over versions on options, thus, the operation interfaces of Eucalyptus and EC2 are similar but not fully compatible. In this evaluation, the operation of EC2 is done with EC2 interface, Eucalyptus with Eucalyptus interface, to evaluated the portability of VM images. The commandline interfaces are almost consistent. They can be integrated by wrapping option differences. As a result, we found that CLC and NC can be rebooted to restore after service down. On the other hand, upon CC failure, all the instances of the cluster are unaccessible, thus it is necessary to reboot the whole Eucalyptus. Eucalyptus lacks failure handling functions, e.g. redundancy, and has less V. WEB APPLICATION PERFORMANCE EVALUATION For the performance evaluation of web application, we evaluated the performance and scalability by giving high load to the web system on Eucalyptus cloud. The evaluation includes OpenPNE access handling performance, login 1018

handling performance, load performance of adding and removing data with database interface. Figure 8 indicates the relationship between the number of application servers and access handling performance and Figure 9 indicates the login/logout handling performance. The survey shows that, when application server and database server are managed as a single instance respectively, the performance of application server that run OpenPNE is the bottleneck. include: short-term development environment, including compile, data manipulation, and application test web system Web system, e.g. enquiry system and small scale SNS Web conference, wiki and groupware e-learning system network transmission as corporate proxy Enterprise proxy Setting up single OS environment for technology education class It should be noted that the quality of Zabbix and virtmanager is sufficient to give monitoring and managing functions to a small-scale cloud system. Eucalyptus has several performance issues, such as storage performance and vulnerability. To be used in a larger system, it is necessary to develop auto-scaling and management interfaces. With these achievements and issues, we are expecting Eucalyptus as an OSS solution to corporate cloud system. Figure 8. The relationship between the number of AP servers and access handling performance REFERENCES [1] http://aws.amazon.com/ec2/ [2] http://www.eucalyptus.com/ [3] http://www.zabbix.com/ [4] http://virt-manager.et.redhat.com/ [5] Daniel Nurmi, Rich Wolski, Chris Grzegorczyk, et al., The Eucalyptus Opne-source Cloud-computing System, 9th IEEE/ACM International Symposium on Cluster Computing and the Grid. pp.124-131. [6] http://www.xen.org/ [7] http://www.linux-kvm.org/page/main_page Figure 9. The login/logout handling performance in combination with verification system As the number of application servers increase, the performance scales up at linear, which implies that it is feasible as a small scale cloud computing, e.g. with one to four database servers, for enterprise use. VI. CONCLUSION We conclude that the cloud system based on Eucalyptus covers basic functionality and can operate flawlessly in a normal situation. From the viewpoints of storage performance and trouble handling, however, it is difficult to implement a large scale of a few hundred servers or a business system, and thus should be limited its use. If limited to small-scale, enterprise private infrastructure for short time use, it can be a good private cloud system. The suggested uses of Eucalyptus private cloud systems 1019