Computing resource optimization with open source virtualization technology

Transcription

1 Computing resource optimization with open source virtualization technology by András Németh Master s Thesis OULU UNIVERSITY OF APPLIED SCIENCES Department of Technology, Communication and Transport Degree Programme in Information Technology 2012

2 ii COMPUTING RESOURCE OPTIMIZATION WITH OPEN SOURCE VIRTUALIZATION TECHNOLOGY ABSTRACT Operating system virtualization techniques allow decouple the operating system from the underlying physical hardware and opens new views to integration and verification teams. In the present thesis work there are two example cases analysed for potential and practical improvements. In the first experiment it is explained how to improve a distributed software source code compiler (build) machine with the help of multiple virtual machines. With this method it will be possible to exceed the maximum computation power compared to one dedicated machine and significantly decrease the required time of software build creation. The second one is a proposal of a virtualized testing environment in which there are several almost identical test hosts execute integration and verification test cases. With this new arrangement it will be possible to manage more easily the test hosts in virtual server environment. Fault tolerance and disk reading performance improvement can be achieved by utilizing distributed disk storage. At the time of writing the thesis it is not yet known whether the given improvement proposals will be implemented and applied in a daily routine at the company (Elektrobit Oulu, Finland). Keywords: virtualization, distributed computing, distributed storage, virtual machine, performance comparison, XEN, XCP

3 iii Table of Contents Abstract ii List of Tables vi List of Figures vii Abbreviations and Acronyms viii 1 Introduction Virtualization: the definition Free/Libre Open Source Brief history of virtualization 2 2 Types of Virtualization Application Level Virtualization Guest Operating System Virtualization Shared Kernel Virtualization Hypervisor Virtualization Storage Virtualization 7 3 Distributed disk arrays and file systems Hadoop Distributed File System Lustre Distributed File System Ceph Distributed File System GlusterFS iscsi ATA over Ethernet 11

4 iv 4 Cloud forming from virtual servers Cloud computing 12 5 Xen Cloud Platform Management Interfaces 13 6 Virtual Machine Cluster Design of Prototype Requirements Resource Usage Considerations Implementation plan Selected virtual server platform Selected Distributed disk solution 17 7 Prototype implementation Installing Xenserver tools Configuration 20 8 Virtual test host using virtual machine array prototype Existing test equipment and resource allocation scenario Current limitations Opportunity of using virtualization Estimated amount of performance improvement Problem Scope Research Approach 21 9 Virtual build machine using virtual machine array prototype Existing scenario Current limitations Opportunity of using virtualization 22

5 v Estimated amount of performance improvement Problem Scope Research Approach Planning * this chapter will be excluded from the final thesis work 23 Appendix A 24 Bibliography 26

6 vi List of Tables 6.1 Memory consumption comparison of linux desktop environments Logical partition requirements and priorities Average disk reading rate comparison on Dell Optiplex 990 at different SATA ports with single and RAID0 configurations Table of Tasks 23

7 vii List of Figures 2.1 Guest operating system virtualization Shared Kernel Virtualization Hypervisor Virtualization Storage Virtualization Virtual Machine Cluster Prototype 16

8 viii Abbreviations and acronyms AMD Advanced Micro Devices AoE ATA over Ethernet BIOS Basic Input/Output System CD Compact Disc CLR Common Language Runtime CP Control Program CPU Central Processing Unit CRUSH Controlled Replication Under Scalable Hashing FLOSS Free/Libre and Open Source Software GbE Gigabit Ethernet GUI Graphical User Interface HDFS Hadoop Distributed File System HVM Hardware Virtual Machine IBM International Business Machines iscsi Internet Small Computer System Interface ISO International Organization for Standardization IP Internet Protocol IT Information Technology I/O Input and Output JVM Java Virtual Machine LAN Local Area Network

9 ix LMDE Linux Mint Debian Edition LTS Long Term Support MBR Master Boot Record MDS Management Data Server OS Operating System OSS Object Storage Servers PC Personal Computer PVHVM Paravirtual on Hardware Virtual Machine RAID Redundant Array of Independent Disks DDR-3 SDRAM Double data rate type three synchronous dynamic random access memory SAN Storage Area Network SATA Serial Advanced Technology Attachment SPOF Single Point of Failure SR Storage Repository SSD Solid-state Drive SW Software VBD Virtual Block Device VIF Virtual (network) Interface VM Virtual Machine VMC Virtual Machine Cluster VMM Virtual Machine Manager XAPI Xen Application Programming Interface XCP Xen Cloud Platform

10 x XFCE Xforms Cool Environment XVP Xen VNC Proxy

11 1 Chapter 1 Introduction In a turbulent economy, virtualization and cloud computing are getting more and more attractive for enterprises because of its convenience and simplicity with the flexibility of traditional computing. Currently, 30 percent of servers are virtualized, and surveys show that by 2012, that number will grow to 50 percent. [8] In this thesis work the advantages of virtualization will be explained and compared to conventional arrangement of computing resources. After a short overview of virtualization history and the summary of the recent virtualization techniques we show how a working prototype of serverless virtual cluster was designed and constructed using commodity hardware. For this prototype cluster there were four identical desktop class computers ordered. Each of the physical computers should be capable of accommodating 1-5 virtual hosts on demand. In both example use cases there are Linux operating systems used. 1.1 Virtualization: the definition Virtualization has an encompassing scope in manner. Numerous definitions can be found from various sources. The one that the author has selected is written by Amit Singh, an IT professional in Virtualization is a framework or methodology of dividing the resources of a computer into multiple execution environments, by applying one or more concepts or technologies such as hardware and software partitioning, time-sharing, partial or complete machine simulation, emulation, quality of service, and many others. [15] Virtualization does not always imply division or partitioning of resources but the opposite. Distributed computing grids appear as one logical entity that can also be interpreted as virtual.

12 2 1.2 Free/Libre Open Source Due to the author is Free/Libre and Open Source Software (FLOSS) advocate, the project merely uses free/libre open source software solutions. We try to highlight that FLOSS is not intrinsically higher or lower quality than proprietary software. It is not inherently more or less secure than its closed source Software (SW) counterpart [12]. The difference resides in the license under which it is made available and the development scheme whether the end user is able to contribute to the product. The FLOSS model allows for able users to view and modify a product s source code. That is often necessary to fully understand their working methods. Using FLOSS and open standards tends to improve interoperability [12] which was considered as a major advantage in the project. Further advantages cited by proponents are expressed in terms of trust, acceptance, teamwork, collaboration and quality [11]. Using open source platform also means that the user is never locked to a proprietary vendor therefore can stay more flexible. 1.3 Brief history of virtualization Virtualization was first implemented in the 60 s by International Business Machines (IBM) corporation. CP-40, the first time-sharing virtual machine/virtual memory operating system provided a Virtual Machine (VM) environment in which multiple instances (up to 14) of client operating systems were running. [23]. The system consisted of a virtualizing Control Program (CP) which created multiple independent VMs. Each virtual machine had its own set of virtual devices, mapped from the system s real hardware environment. CP segregated complex system problems from a single user application. Isolating users from each other improved system stability as a bug in one user s software could not crash another user s application nor the underlying CP. The platform was made generally available to IBM customers in source code form later in [23] In the 70 s and 80 s improved versions of CP-40 virtualization platforms are introduced in numerous large IBM mainframes. Since mainframes were expensive resources at

13 3 the time, they were designed for partitioning as a way to fully leverage the investment.[13] In the 90 s inexpensive x86 server and desktop deployments led to new infrastructure challenges. More and more Linux and Windows server operating systems became available running on x86 processor architecture. In 1999, VMware introduced the first x86 virtualization product, VMware Virtual Platform. Unlike mainframes, x86 machines were not designed to support full virtualization. It was achieved by complex software techniques. Since 2006 Intel and Advanced Micro Devices (AMD) processors have hardware virtualization capability. These hardware features differ between Intel and AMD processors. Intel named its technology VT-x; AMD calls theirs AMD-V. On many systems, the hardware virtualization features first need to be enabled in the Basic Input/Output System (BIOS) before VM can use them. The Central Processing Unit (CPU) flag for AMD-V is svm and for VT-x is vmx. This can be displayed in Linux operating systems via /proc/cpuinfo. andras@desktop ~ $ cat /proc/cpuinfo grep flags flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save andras@desktop ~ $

14 4 Chapter 2 Types of Virtualization When selecting the best method of implementing virtualization it is essential to have a clear understanding of the different virtualization solutions which are currently available. 2.1 Application Level Virtualization In application level virtualization VM runs as a single process inside host Operating System (OS). The purpose of application virtual machine is to provide platform-independent programming environment that allows a program to execute in the same way on any platform [9]. This type of VM has become popular with the Java programming language, which is implemented using the Java Virtual Machine (JVM). In.NET framework the similar purpose process VM is called Common Language Runtime (CLR). Process VMs are always implemented using an interpreter. Application level virtual architecure allows creating platform independent and highly portable applications. The application using these kinds of VMs have naturally lower performance than compiled programming languages. 2.2 Guest Operating System Virtualization Guest operating system virtualization is also called software-based virtualization. This is the most simple and the easiest concept of all OS level virtualization methods. It does not require CPU virtualization support. The physical host computer runs a standard unmodified operating system such as Windows, Linux, Unix or MacOS X. Guest operating systems are created and ran within a virtualization application. The virtualization application is responsible for starting, stopping and managing each virtual machine and controlling physical hardware access. The virtualization application reads the executing guest OS CPU operation calls and replaces each privileged instruction with safe emulations. The most commonly known open source virtualization application is Virtualbox.

15 5 Figure 2.1: Guest operating system virtualization 2.3 Shared Kernel Virtualization Shared Kernel Virtualization is also known as system level or operating system virtualization that is available on Linux and Unix based operating systems. This type of virtualization is made possible by the ability of the kernel to dynamically change the current root file system. This concept requires the guest operating system to be compatible with the shared kernel version. For example a 32bit guest operating system architecture will not be accessible by using a 64bit shared kernel version. With the help of chroot command it is posssible to change root file system from a host OS using a shared kernel. On many systems, only the super-user ( a user with root privileges) can do this. Chroot can also be used to fix problems when the OS does not boot correctly because of problems in rootfs or Master Boot Record (MBR). Major web hosting providers are using shared kernel virtualization for years so that customers get their own virtual server for their web hosting needs. The customers don t know that the system is virtual, nor can they contact the host system through their VM [10].

16 6 Figure 2.2: Shared Kernel Virtualization 2.4 Hypervisor Virtualization Hypervisor (also called as type 1 Virtual Machine Manager (VMM)) is a software layer that runs directly on host computer replacing the operating system. In this case, the VMM itself is the minimalistic OS. The hypervisor is the interface for all hardware request such as CPU, Input and Output (I/O), and disk for the guest operating systems [21]. It is so named because it is conceptually one level higher than a supervisory program or operating system. Hardware Virtual Machine (HVM) term is used do describe the guest operating system that is running in hardware-assisted virtualization environment. This technique requires CPU virtualization extensions (Intel VT or AMD-V). HVM guests do not require special kernel, for example native windows operating systems can be used as HVM guests [28]. Fully virtualized guests are usually slightly slower than paravirtualized ones, but HVM guests can use special paravirtual device drivers to bypass the emulation for disk and network I/O. These Paravirtual on Hardware Virtual Machine (PVHVM) drivers provide much faster disk- and network I/O performance.

17 7 Figure 2.3: Hypervisor Virtualization Paravirtualization is another technique that relies on hypervisor virtualization but does not require virtualization support from the host CPU. The guest OS is aware of the hypervisor therefore only modified guest operating systems can be loaded. Linux kernel version and above have the Xen PV guest support and includes all the necessary patches needed to be used as PV guests. [28] Microsoft Windows requires a HVM Guest and can not be used in paravirtualized environment. Hypervisor solutions are available from different vendors and sources such as: XenServer, VMware ESX/ESXi, and Microsoft Hyper-V hypervisor. The author has selected Xen open source hypervisor with hardware virtual machine solution for the prototype implementation. In the upcoming chapters Xen server will be explained in more detail and will be used in examples. 2.5 Storage Virtualization Storage virtualization is a concept in which storage systems use special tools to enable better functionality and more advanced features within a storage system [22]. The main feature of

18 8 storage virtualization is the abstraction of the logical and physical location of the data. One of the major benefits is the non-disruptive data migration when the data can be freely moved or replicated without affecting the operation of any client. Concurrently performed disk operations can significantly improve the I/O performance while the utilization of physical resources remain load-balanced. Different solutions are available based on the needs for availibility, I/O performance, search and indexing and for a combination of these. Figure 2.4: Storage Virtualization A simple version of storage virtualization is considered in the design of the virtual machine cluster. Some of the most common practically used distributed disk arrays are explained in the following chapter.

19 9 Chapter 3 Distributed disk arrays and file systems As a response to demand for data intensive file systems, several distributed file systems has been developed in recent years. It is envisioned a further large scale increase in the use of parallel programming tools, scientific applications, data mining, etc. [30] 3.1 Hadoop Distributed File System The Hadoop Distributed File System (HDFS) is a highly fault-tolerant distributed file system that is designed to be deployed on low-cost hardware. [31] Hadoop is an open source Apache TM project that is used by a wide variety of companies and organizations including Amazon, Google ans Yahoo [1]. Hadoop can be configured to be working in standalone mode, but extendable to extreme large clusters with thousands of nodes. A HDFS cluster consists of two node types. NameNode manages the file system metadata and DataNodes that store the actual data. Hadoop has software framework (MapReduce) for writing applications which can process vast amount of data (multi-terabyte data-sets) in parallel manner on clusters of thousands of nodes. Hadoop is not a drop-in replacement for a database or Storage Area Network (SAN) filesystem because of its significantly higher response time. 3.2 Lustre Distributed File System Similarly to Hadoop, Lustre also has two server node types, namely Management Data Server (MDS) and Object Storage Servers (OSS). On client side after the necessary kernel modules loaded it is possible to mount the lustre cluster. The system allows multiple clients accessing the same files concurrently, and all the clients see consistent data at all times [24]. Expanding the array is possible on the fly without interrupting any operations. Several Linux distributions include Lustre file system libraries and utilities in their software

20 10 repositories. Many of the users of Lustre do not recommend to others because it easily breaks down in all kinds of situations [4]. 3.3 Ceph Distributed File System Ceph is a distributed network storage and file system with distributed metadata management [5]. In the file system there is a cluster of metadata servers, which manages the namespace (file names and directories) while coordinating security, consistency and coherence. [32]. The minimal system does not require metadata server and have at least two MDSs for data replication. The allocation list is predictably striped across storage devices using a distribution function called Controlled Replication Under Scalable Hashing (CRUSH). This method eliminates the need to maintain object lists and look-up tables. Ceph client is merged to Linux kernel since version [7]. 3.4 GlusterFS GlusterFS is an open source, highly scalable clustered file system. GlusterFS can be flexibly combined with commodity physical, virtual and cloud resources to deliver highly available and performant storage with relatively low cost. GlusterFS is suitable for public and private cloud environments. [7] Unlike other cluster file systems, GlusterFS does not use centralized meta-data server. This feature makes it simple and easy to deploy in prototype environments. The author has selected this filesystem to be used in the current project. 3.5 iscsi There are several existing software solutions that was designed help building a distributed disk array between computers using Ethernet interface. Internet Small Computer System Interface (iscsi) is a network storage protocol over the Internet Protocol (IP). The protocol allows to use SCSI commands over an IP network instead of a costly Fibre Channel. The clients (called initiators) can send SCSI commands to block storage devices (targets) on

21 11 remote servers [20]. iscsi uses a client-server architecture and usually only one client has access to the remote block storage. A server that makes targets available only needs free partition to be available for export, and it does not need to be a SCSI disk. The technique may be combined with clustering solutions by exporting logical volume(s) and/or Redundant Array of Independent Disks (RAID). Open-iSCSI is an open source project that provides multi-platform implementation of iscsi [17]. The binaries are available for all major Linux distributions and can be easily installed through package manager. 3.6 ATA over Ethernet ATA over Ethernet (AoE) is a network protocol that was designed and optimized for highperformance access of Serial Advanced Technology Attachment (SATA) storage devices over Ethernet. [2] AoE is a layer 2 protocol which makes it fast and lightweight. The protocol is non-routable, so our distributed disk array could not be extendable over different subnetworks. On Linux it is relatively simple to implement such a solution using aoetools [3] and vblade software packages. Vblade is a program that makes a seekable file available over an ethernet local area network (LAN) via the AoE protocol. The seekable file is typically a block device or partition like /dev/md0 or /dev/md0p1. When vblade exports the block storage over AoE it becomes a storage target. Another host on the same Local Area Network (LAN) can access the storage if it has a compatible AoE kernel driver. Similarly to iscsi, only one client has access to the remote block storage.

22 12 Chapter 4 Cloud forming from virtual servers Once the virtual OS instances running on virtual servers, the question arises how to merge and manage these server instances through one management console. The question is rather complex when we start thinking more deeply what kind of support the imagined management framework should provide. Clearly we need a higher level solution for node management, storage management, image repository management with proper authentication. The solution can be found in the cloud. 4.1 Cloud computing Cloud computing is a model for access to a shared pool of configurable computing resources [16]. The term cloud is used as a metaphor for the Internet in network diagrams and it symbolizes an abstraction of the underlying infrastructure it represents. The cloud model is composed of the following deployment models: Private cloud is operated and used exclusive by a single organization. Community cloud is for exclusive use by a specific community of consumers. Public cloud is provisioned for open use by the general public. Hybrid cloud is a composition of two or more above mentioned cloud models. Openstack is one of the available open source provisioning systems that helps build and manage large clusters.

23 13 Chapter 5 Xen Cloud Platform The Xen Cloud Platform (XCP) open source server virtualization and cloud computing platform. It is derived from Citrix Xenserver proprietary product [27] and built on the top of Xen hypervisor. The functionality is so much similar to Citrix XenServer than all the XenServer manuals also apply to XCP. For instance XenServer 6.0 documents apply XCP 1.5 [25]. The first stable version (1.0) of XCP was announced in At the time of writing this paper, the latest version is 1.5 which was released in 2012 and CentOS 5 is serving as the Dom0. XCP was selected to be utilized in the Virtual Machine Cluster (VMC) realization. 5.1 Management Interfaces XCP includes Xen Application Programming Interface (XAPI) toolstack which allows managing the server either by using command line or Graphical User Interface (GUI) or web management tools. Xe (xe) is a powerful command-line interface which talks to both hosts and resource pools over https, invoking XenAPI operations. Commands may be executed both from within DOM0 and from remote hosts. Tab completion is available which is increasing usability and speed. Xe enables writing of scripts for automating tasks and allows integration into an existing Information Technology (IT) infrastructure. Xe is installed by default on XCP server. A stand-alone remote version is also available for Linux [6]. Openxenmanager [18] is a multiplatform tool written in python with graphical interface to manage XenServer / XCP hosts over the network. OpenXenManager is an opensource clone of Citrix XenCenter windows application. Xen VNC Proxy (XVP) [14] is a suite of open source programs for management of virtual machines running on XenServer and XCP developed by Colin Dean. The console allows to operate and access virtual machines through a web browser.

24 14 Chapter 6 Virtual Machine Cluster A serverless cluster of computers could provide high scalability to perform the required tasks described earlier. In this chapter the design of such a cluster is presented. 6.1 Design of Prototype Requirements REQ01: The designed system should be capable of providing up to 30 virtual machine instances that can be used for executing integration and verification test scripts. REQ02: The designed system should be easily integrated to existing SW and HW infrastructure and should be compatible with it. REQ03: The designed system should be easily extendible and scalable on demand. REQ04: The designed system should be easily reconfigurable for various kind of tasks. E.g. SW build environment. REQ05: Assessment of potential Single Point of Failure (SPOF) should be made. The number of critical components should be reduced or eliminated. Fault tolerance should be provided Resource Usage Considerations Due to the fact that the VMC servers are built using standard desktop class computers, special care should be taken when selecting the SW components. The limited memory and CPU processing capacity compel us to seek for a less resource intensive, lightweight environment. A simple desktop environment comparison was made by using gnome-system monitor to compare memory consumption of different linux operating systems. The 64-bit version

25 15 OS version (64-bit) Desktop Environment Used memory (MiB) Linux Mint LMDE (12.04) XFCE Linux Mint LMDE (12.04) MATE/Cinnamon Ubuntu Desktop LTS Unity Table 6.1: Memory consumption comparison of linux desktop environments of installer ISO disk image was downloaded and booted in Virtualbox application. Only the default loaded services were running and the gnome-system monitor application which displayed the actual resource usage. The comparison showed that the less resource hungry desktop environment is XFCE using about 38% of memory compared to Unity which is the default environment in Ubuntu release Similar comparison was made in 2010 by Phoronix with similar results [19]. The same study also showed that there were not any significant power consumption differences between the tested desktop environments Implementation plan The demand for productive virtual testing environment is progressively growing. We have to schedule the implementation steps synchronously with the needs. In the early phase there is no specific need for disk redundancy options and the HW utilization does not require performance adjustments. These can be added later without disturbing the existing virtual configurations. Phase 01: Collect and define internal requirements for the future system. Study existing practical virtualization technologies and tools to be used and select the most suitable one that can be integrated easily to the existing infrastructure. Phase 02: Order the selected HW resources. Prepare the environment for quick deployment when the HW resources are available. Phase 03: Implement a standalone virtual server for reference using only local disk. Create first instance of general virtual machine for templating base. This VM can be used for demo purposes.

26 16 Phase 04: Use template VM to create a production version of VM and clone multiple instances according to the needs. At this phase it is still the standalone virtual server is in use with local disk repository. Phase 05: Install virtual servers on all the physical machines. Build distributed file system. Do performance measurements on distributed volumes. Tune configuration parameters if necessary. Phase 06: Clone as many production VMs as needed using the distributed filesystem. Do performance measurements on production VMs and tune configuration parameters if necessary. Phase 07: Finalize the environment, extend the VMC with new nodes on demand. Figure 6.1: Virtual Machine Cluster Prototype

27 17 Chapter 7 Prototype implementation Four pieces of Dell Optiplex 990 desktop Personal Computers (PCs) were ordered for building the VMC system. The PCs were equipped with Intel Core i7 quad core processors, 8GB of Double data rate type three synchronous dynamic random access memory (DDR-3 SDRAM), 500GB SATA Hybrid Hard Drive and three pieces of Gigabit Ethernet (GbE) network interface. Additionally, two OCZ Vertex-4 Solid-state Drive (SSD) were installed in each PC. After installing the SSDs, it was observed that only one of the four port is supporting SATA-3 (6Gbps) transfer data rate. RAID0 configuration of using two similar SSDs resulted 8.2% reading speed improvement. Configuring SSD and Hybrid hard disk performed even worse than single SSD drive. 7.1 Selected virtual server platform The XCP open source server virtualization and cloud computing platform was selected and deployed on each of the four computer in the cluster. XCP is easily installable from ISO image and includes all of the necessary components for the target cluster. It includes the Xen hypervisor, lightweight dom0 privileged domain and powerful command line control interface (xe). The Xen Cloud Platform 1.5 Beta was available from Xen internet page [26]. The size of the image was 362MiB. The installation was carried out from Compact Disc (CD) media and took approximately half an hour. XCP uses the whole physical disk and does not allow to leave another operating system on a different partition. The dom0 physical disk footprint is less than 200MiB. This is enough to accomodate the hypervisor, necessary xen services, command line tools and a minimalistic consol user interface. The rest of the physical hard disk space can be used to store virtual disks and/or ISO images in a Storage

28 18 Test 1 Port number Average Read Rate (MiB/sec) Disk type OCZ-Vertex OCZ-Vertex OCZ-Vertex OCZ-Vertex4 Test 2 Port number Average Read Rate (MiB/sec) Disk type OCZ-Vertex GB SATA HDD OCZ-Vertex4 3 Not Tested CDROM Test 3 - RAID0 configuration Port number Average Read Rate (MiB/sec) Disk type 0 OCZ-Vertex4 Total: OCZ-Vertex4 Test 4 - RAID0 configuration Port number Average Read Rate (MiB/sec) Disk type 0 OCZ-Vertex GB SATA HDD Total: OCZ-Vertex4 Table 7.1: Average disk reading rate comparison on Dell Optiplex 990 at different SATA ports with single and RAID0 configurations

29 19 Logical partition requirements and priorities Purpose Size approx. Fault tolerance R/W performance Boot (local) 4x10GB YES YES Virtual machine loaded 4x100GB NO YES Virtual machine archive Maximum available YES NO Shared working area Maximum available NO YES Table 7.2: Logical partition requirements and priorities Repository (SR). 7.2 Selected Distributed disk solution The option of having distributed array of disks provide the most flexible and scalable solution for the virtual machine cluster. One additional dedicated GbE network interface card is reasonably fast for the disk array. There will be no other traffic on that specific interface. 7.3 Installing Xenserver tools Installing XenServer Tools (PV Drivers) provide higher performance network and disk I/O operations without the overhead of traditional device emulation [29]. XCP includes the xenserver tools ISO image (xs-tools.iso) which can be easily loaded and installed onto guests. We have performed the Xenserver tool installation on a linux HVM guest (Linux Mint 13 MATE). Several additional VM features unveiled that were listed as <not in database> before (guest OS version, IP address). VM allowed operation list now includes pool_migrate, suspend and checkpoint. The number of Virtual Block Device (VBD) and Virtual (network) Interface (VIF) devices have increased. For the full list of additional features, consult Appendix A on page 24. On the general tab in the properties pane within OpenXenmanager the message Tools not installed has been replaced by Optimized (version 1.4 build 53341) and the OS version is also displayed. Machine suspend and resume operations were tested and worked through OpenXenmanager.

30 7.4 Configuration 20

31 21 Chapter 8 Virtual test host using virtual machine array prototype 8.1 Existing test equipment and resource allocation scenario Current limitations Opportunity of using virtualization Estimated amount of performance improvement 8.2 Problem Scope 8.3 Research Approach

32 22 Chapter 9 Virtual build machine using virtual machine array prototype 9.1 Existing scenario Current limitations Opportunity of using virtualization Estimated amount of performance improvement 9.2 Problem Scope 9.3 Research Approach

33 23 Chapter 10 Planning * this chapter will be excluded from the final thesis work This is a simple thesis writing plan and a list of tasks. Theoretical thesis preparation timeline Task Time to Finish Notes Study Current environment 1-2 weeks Done Present proposal at company 1 day Done More detailed explanation at company 1 day Done Prepare abstract, send to OAMK 2 days Done Select prototype hardware and order 1 day Done Research 1 day Literature search at library Study literature Distr. Disk 2-3 weeks on week 18-21, 2012 Study literature virtual machines 3-4 weeks on week 22-26, 2012 Table 10.1: Table of Tasks

34 24 Appendix A Virtual machine parameter list before installing Xenserver tools: [root@fidv ~]# xe vm-param-list uuid=1e16e81e fb2-3f1e-64c1dcd6919f uuid ( RO) : 1e16e81e fb2-3f1e-64c1dcd6919f name-label ( RW): Linux Mint 13 MATE-virtual01...lines removed... allowed-operations (SRO): changing_dynamic_range; changing_vcpus_live; hard_reboot; hard_shutdown; clean_reboot; clean_shutdown; pause; snapshot...lines removed... allowed-vbd-devices (SRO): 2; 3 allowed-vif-devices (SRO): 0; 2...lines removed... os-version (MRO): <not in database> PV-drivers-version (MRO): <not in database> PV-drivers-up-to-date ( RO): <not in database> memory (MRO): <not in database> disks (MRO): <not in database> networks (MRO): <not in database> other (MRO): <not in database> live ( RO): <not in database> guest-metrics-last-updated ( RO): <not in database>...lines removed... Virtual machine parameter list after installing Xenserver tools: [root@fidv opt]# xe vm-param-list uuid=1e16e81e fb2-3f1e-64c1dcd6919f uuid ( RO) : 1e16e81e fb2-3f1e-64c1dcd6919f name-label ( RW): Linux Mint 13 MATE-virtual01...lines removed... allowed-operations (SRO): changing_dynamic_range; pool_migrate; changing_vcpus_live; suspend; hard_reboot; hard_shutdown; clean_reboot;

35 25 clean_shutdown; pause; checkpoint; snapshot...lines removed... allowed-vbd-devices (SRO): 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15 allowed-vif-devices (SRO): 0; 2; 5; 6...lines removed... os-version (MRO): name: Linux Mint 13 Maya; uname: generic; distro: linuxmint; major: 13 PV-drivers-version (MRO): major: 1; minor: 4; micro: 90; build: PV-drivers-up-to-date ( RO): true memory (MRO): disks (MRO): networks (MRO): 0/ip: other (MRO): platform-feature-multiprocessor-suspend: 1; feature-balloon: 1 live ( RO): true guest-metrics-last-updated ( RO): T13:42:36Z...lines removed...

36 26 Bibliography [1] Alphabetical list of institutions that are using hadoop for educational or production uses. Retrieved 02 May [2] Ata over ethernet. over Ethernet, Retrieved 01 May [3] Ata over ethernet tools. Retrieved 01 May [4] Best distributed filesystem for commodity linux storage farm. com/questions/269179/best-distributed-filesystem-for-commodity-linux-storage-farm, Retrieved 11 May [5] Ceph documentation. Retrieved 11 May [6] Citrix xenserver 6.0 administrator s guide /1.0/en gb/reference.html, Retrieved 09 June [7] Client merged for ceph. Retrieved 03 May [8] Cloud computing & virtualization: Hot trends organizations can t ignore. cloudcomputing.sys-con.com/node/ , Retrieved 30 April [9] Cloud computing & virtualization: Hot trends organizations can t ignore. en.wikipedia.org/wiki/virtual machine#process virtual machines, Retrieved 13 May [10] Comparing virtualization technologies. aspx?p= &seqnum=5, Retrieved 29 May [11] The gnu manifesto - gnu project - free software foundation (fsf). gnu/manifesto.html, Retrieved 12 May 2012.

37 27 [12] A guide to open source software for australian government agencies (second edition). html, Retrieved 12 May [13] History of virtualization. Retrieved 30 April [14] Introduction to the xvp suite of programs. Retrieved 09 June [15] An introduction to virtualization. virtualization/, Retrieved 31 May [16] The nist definition of cloud computing, national institute of standards and technology. Retrieved 18 May [17] Open-iscsi project: Open-iscsi - rfc3720 architecture and implementation. open-iscsi.org/, Retrieved 03 June [18] Openxenmanager s official wiki. Retrieved 09 June [19] Power and memory usage of gnome, kde, lxde and xfce. scan.php?page=article&item=linux desktop vitals&num=1, Retrieved 26 May [20] A quick guide to iscsi on linux. Retrieved 03 June [21] Server virtualization with the xen hypervisor. WhatisXen.pdf, Retrieved 12 May [22] Storage virtualization. virtualization, Retrieved 14 May 2012.

38 28 [23] Timeline of virtualization development. of virtualization development, Retrieved 30 April [24] Understanding lustre. HTML/ UnderstandingLustre.html, Retrieved 03 May [25] Xcp - manuals and documentation. Retrieved 09 June [26] Xen - xcp download. Retrieved 26 May [27] Xen cloud platform project. Retrieved 09 June [28] Xen overview. Overview, Retrieved 20 May [29] Xenserver tools. gb/guest.html# windows pvdrivers, Retrieved 09 June [30] Wittawat Tantisiriroj Bin Fan et al. Diskreduce: Raid for data-intensive scalable computing. [31] Dhruba Borthakur. Hdfs architecture guide. current/hdfs design.html, Retrieved 02 May [32] Sage Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. Ceph: A scalable, high-performance distributed file system. November 2006.