Table of Contents Objective...3 Scope...3 Definitions...3 Initial Criteria...4 Generic...4 Support Services...4 Features...4 Systems Administration...5 Additional Criteria...5 Product Discovery...5 Initial products list...5 Additional products reviewed...5 Bibliography...6 Products Tested...7 Testing Discussion...7 Criteria Weights and Product Scores...8 Criteria Comparison Discussion...9 Generic...9 Security...9 Cost...9 Hardware requirements...9 Support Services...9 Documentation...9 Training opportunities...10 Technical support...10 Features...10 Guest OS support...10 Live migration...11 Cluster of hosts...11 Shared storage...11 Automatic disaster recovery...11 Systems Administration...12 Ease of management...12 Remote access...13 Scripting support...13 Resource monitoring...13 Performance...14 1
Budget Considerations...15 2
Objective Compare virtual server software products to recommend a common virtual server environment for ASC's linux and windows servers. Scope A virtual server solution for linux and microsoft windows servers managed by the network team systems administrators group. Discussion: The Banner team selected a VMware ESX environment for their needs. It was decided the Banner environment was outside the scope of this trade study other than seeking input for testing and comparison purposes. The network team systems administrators have been running the free VMware server product for some time after an initial virtualization peer review. This trade is a follow up to the initial peer review with a broader scope of application including all network team servers running linux and Windows operating systems. Definitions Virtualization In computing, to provides a logical rather than physical view of data, computing power, storage capacity, and other resources. Virtual Machine Software that mimics a hardware device, in this case a complete computer system. Hypervisor A base operating system which is dedicated to running virtual servers. Host Server running a hypervisor. Guest A virtual machine running on a host. Paravirtualization A virtualization technique that presents the abstraction of virtual machines with a software interface that is similar but not identical to that of the underlying hardware. This requires operating systems to be explicitly ported to run on top of the virtual machine monitor (VMM). SAN: (Storage Area Network) Is a high speed sub network of shared storage devices. iscsi: A network storage protocol that enables the sending of SCSI commands over a TCP/IP network. Primarily used in Storage Area Networks. 3
NFS: Network File System. A protocol created by Sun Microsystems that allows a machine to "mount" part of all of the disk of another machine over a LAN. HA: (High Availability) A computing system that is designed for continuous operation in the event of a failure of one or more components. Initial Criteria Generic Security How secure is the environment? Is there a proven security track record? Cost How affordable is the product? Hardware requirements How much hardware is required? Less hardware and associated expense is better. Support Services Documentation How good is the documentation for use by our systems administrators? Training opportunities Do adequate training opportunities exist for our systems administrators? Technical support How good is the quantity and quality of 3rd party support for the product? Features Live migration Can a guest be moved from host to host with no downtime, how well does it work Cluster of hosts How well is the VM host cluster of resources implemented Shared storage How well does the product inter operate with open standard shared storage back ends? Automatic disaster recovery How good is automatic recovery from a guest VM failure 4
Systems Administration Ease of management How easy is the product to maintain? Are patches and version upgrades easy to apply? Remote access How good is remote access to management tools Scripting support How good is the scripting support for VM management Resource monitoring How easy can the host cluster resources be monitored? How easy can guest resources usage be monitored? Additional Criteria Guest OS support How well are our target guests OS's supported? Note: Guest console access is discussed within Ease of Management Guest console access How easy is it to access guest server consoles? Performance How well do guest's perform? Do guests exhibit any bottlenecks? Product Discovery Initial products list VMware ESX Server Open Source XEN Citrix Xenserver Microsoft HyperV Additional products reviewed Linux KVM 5
Bibliography The 12 References listed in the virtualization peer review. 2007 Site visit, Colorado College VMware Cluster Q&A. 2007 Dell/EMC/VMware sales presentation Q&A. 2008 ASC Banner VMware demonstration Q&A. 2008 Microsoft Management Summit Q&A. Microsoft Hyper V website http://www.microsoft.com/windowsserver2008/en/us/virtualization consolidation.aspx Hyper V supported guests http://www.microsoft.com/windowsserver2008/en/us/hyperv supported guest os.aspx Virtualization Review "Hyper V: Under the Hood" http://virtualizationreview.com/features/article.aspx?editorialsid=2477 Citrix Xenserver product documentation CS/Architecture/virtual servers/xen citrix Citrix Xenserver webcast demonstration http://www.citrix.com/xenserver/demo Citrix Xenserver supported guest operating systems list http://www.citrix.com/english/ps2/products/feature.asp?contentid=1297950 Linux KVM main website http://kvm.qumranet.com/kvmwiki/ Linux KVM guest support status page http://kvm.qumranet.com/kvmwiki/guest_support_status 2007 Ottowa Linux Symposium, "kvm: the Linux Virtual Machine Monitor" http://ols.108.redhat.com/2007/reprints/kivity Reprint.pdf KVM/Citrix/VMware benchmark attempt http://www.brianmadden.com/blog/brianmadden/qumranet testing day one a new method forbenchmarking Also XenDesktop hangs and VMware withdraws CNet News "KVM steals virtualization spotlight" http://news.cnet.com/kvm steals virtualization spotlight/2100 7344_3 6161804.html Phronix "Ubuntu 8.04 KVM Benchmarks" http://www.phoronix.com/scan.php?page=article&item=ubuntu_virt_benchmarks&num=1 Secunia security advisory database: http://secunia.com/ 6
Products Tested VMware ESX server Citrix Xenserver Linux KVM Testing Discussion To test VMware ESX Server we used the existing Banner installation comprised of Dell 2950 servers and an Aberdeen iscsi back end SAN. We did some tests using existing installations of the free VMware server product on Dell 2950 server hardware. Citrix XEN Server was tested via an evaluation copy on a Dell 2950 server. Linux KVM initial testing took place on Dell Optiplex workstations and a Lenovo Thinkpad laptop. Some tests used an iscsi target served via a VMware linux guest. Final testing hardware included two Dell 2950 servers and an Aberdeen storage server running gentoo linux with base line linux iscsi, OCFS2, and NFS software. The linux KVM test environments ran Ubuntu linux. Both Microsof Hyper V and Citrix Xenserver were removed from consideration early in the testing process due to a documented lack of needed guest OS support. Microsoft's product was further affected by being under beta testing status. The open source Xen product was removed from consideration when Citrix Xenserver was determined inadequate. At that point a decision was made to investigate the open source linux KVM product instead. 7
Criteria Weights and Product Scores weights Score (1-5) Weighted Score (1-5) Weigthed Criteria VMware ESX VMware ESX VMware ESX VMware ESX VMware ESX linux KVM linux KVM linux KVM linux KVM linux KVM Generic Security 7 3 2 3 4 84 4 4 4 4 112 Cost 5 2 3 1 3 45 4 5 5 5 95 Hardware requirements 5 2 3 4 3 60 4 5 4 4 85 Total 189 292 Normalized 55.59 85.88 Support Services Documentation 8 4 4 4 4 128 3 1 4 3 88 Training opportunities 3 4 4 5 4 51 1 1 3 2 21 Technical support 4 4 4 4 5 68 3 2 3 3 44 Total 247 153 Normalized 82.33 51 Features Guest OS support 7 2 4 5 4 105 5 5 5 4 133 Live migration 7 4 5 3 4 112 4 4 5 4 119 Cluster of hosts 3 4 5 4 4 51 3 3 3 4 39 Shared storage 9 4 3 4 4 135 3 3 4 4 126 Automatic disaster recovery 9 4 3 4 4 135 3 3 3 3 108 Total 538 525 Normalized 76.86 75 Systems Administration Ease of management 7 3 4 4 4 105 4 2 4 2 84 Remote access 6 2 4 4 4 84 4 3 5 4 96 Scripting support 5 2 2 3 4 55 5 5 4 4 90 Resource monitoring 11 3 5 4 5 187 4 4 4 4 176 Performance 7 3 3 4 5 105 4 5 5 4 126 Total 536 572 Normalized 74.44 79.44 Normalized Total 289.22 291.33 VMware: 289.22 linux KVM: 291.33 VMware wins: Support services linux KVM wins: Generic (Security/Cost/Hardware req's) Products tie: Features and Systems Administration 8
Criteria Comparison Discussion Generic Security Security alerts exist for both VMware's products and the linux KVM. Secunia reports 67% of VMware ESX server vulnerabilities as being attackable from remote sources for 2007 and 88% from remote sources for 2008, with 33% highly critical in 2007 and 63% highly critical for 2008. Secunia's KVM stats include 2 notices, all from 2008, none are remote and none are in the highly critical or greater range. KVM is an official part of the Linux kernel. The stats for the linux kernel for 2007 show 22% of advisories as remote, and none as highly critical or above. For 2008 17% are remote and none highly critical or above. VMware best practices include housing the VMware management and storage environments on inaccessible private networks but interfaces need to be available on VMware hosts to allow for Internet accessible guests. KVM makes no recommendations, but best practice for storage networks is to compartmentalize them on private networks. Cost The linux KVM product is free of charge. Paid support is available via the purchase of a commercial linux edition support contract. The target environment for our considered KVM deployment does not include a commercial linux edition. VMware ESX server cost was determined to be a minimum of $12,000/yr for our target environment. Further investigation reveals VMware ESX server licensing to be 3 times the cost of the underlying server hardware for our target environment. See the Budget Considerations section for details. Hardware requirements All products except Microsoft's Hyper V appeared to support our target hardware level on a guest vm per host server basis. VMware ESX server features over subscription of some host resources, this feature was deemed unnecessary for our needs and we did not test it's behavior. We have run oversubscribed on CPU's, (more guest CPU's than a host's CPU cores), with both the free VMware server product and linux KVM. Support Services Documentation Citrix Xen and VMware have collections of PDF manuals, and online forums. Linux KVM has various 9
levels of documentation ranging from man pages, vendor produced how to documents, forums, wiki's mailing list etc.. While the initial set of VMware documentation is very complete, some features or nuances of running the environment are still only found by digging through forum posts and web searches. The linux KVM documentation is quite short, reflecting the simplicity of the environment. As with VMware a good amount of searching is required to uncover yet to be completely documented features. An example, VMware has an undocumented e1000 virtual nic device driver, and while KVM documentation shows 10 different virtual nic drivers, 2 others show up via a command line query for available nic drivers. Training opportunities A lot of VMware training is available for systems administrators. The closest thing to linux KVM specific training are talks and sessions at various conferences. For linux KVM the level of training required for ASC to administer the system has already been surpassed in establishing a working test environment. Technical support VMware technical support is good. Citrix xenserver support is lousy. Paid linux KVM support is available if running a vendor supported linux, commercial linux product, or via contract with consultants. Thus far our experiences with paid linux support is few and far between for any product. In the case of linux KVM paid support is most likely not needed, traditional open source community support appears adequate. Features Guest OS support Both Citrix Xenserver and Microsoft Hyper V had guest support problems. Xenserver would not run our target guest Linux OS's, and Microsoft's documentation claimed not to either. We eliminated both products from further investigation when this was discovered. Both VMware ESX and linux KVM will run all guest OS's we are targeting. While VMware does not officially support some of our target work load, some of the guest software vendors do not support running virtualized either. One interesting indicator of the level of guest OS support, linux KVM is being used by CISCO to run CISCO IOS for some of their new switches. 10
Live migration Both VMware ESX and linux KVM support the transfer of a running vm between physical host machines. The operation requires some type of shared storage accessible by each host, on which the guest virtual machine is stored. Live migration in our test environments on both VMware ESX and linux KVM worked well. VMware ESX requires special configuration of guests to migrate guests between hosts running different CPU families, including different versions of 64 bit Intel CPU's (5300 and 5400 series). Linux KVM allows live migration between differing Intel CPU families with no extra configuration needed, testing included live migration between a Centrino duo and Pentium D series host. Cluster of hosts VMware implements different types of host clusters via their management interface. VMware host clusters can be configured for a logical view, disaster recovery, or resource sharing / live migration. Linux KVM has no built in abstraction of a cluster of hosts other than support for migration of a running vm between servers. With shared storage and standard linux HA software linux KVM can be configured to support disaster recovery. No automatic moving of guests between hosts based on resource use levels is built into linux KVM, however this functionality was easily scripted during testing. Extra care is needed when adding new host servers to a VMware cluster, to avoid issues with CPU family compatibility such as those mentioned previously under live migration. Shared storage VMware allows access to iscsi or NFS for shared storage. NFS was not tested from VMware. VMware uses it's own proprietary cluster file system. Open source cluster file systems require SCSI 3 persist/reserve support on the shared iscsi target, our SAN does not support the SCSI 3 command set. For linux KVM shared storage and live migration testing we used NFS. VMware's configuration to access iscsi was a bit tricky. Linux access to iscsi or NFS is straightforward, but care must be taken with NFS configuration to maintain access when a SAN HA fail over occurs. Automatic disaster recovery VMware ESX includes the ability to automatically reboot a downed guest on another host. Host CPU family compatibility is a concern with VMware ESX server guests for a proper reboot to occur. A linux KVM guest can be restarted on another host via standard linux HA mechanisms. In each case shared storage is required. When shared storage is disconnected or lost, VMware virtual machines required a restart after storage availability was restored. Linux KVM guests with a lost shared storage connection continued to run and became immediately usable when the storage connection was restored, no guest reboot was required. 11
Systems Administration Ease of management Patches and Upgrades VMware patch sets for ESX server are not reversible. As this document was being composed reports of a licensing bug in the latest VMware patch set surfaced, the bug caused all VMware guests to be unable to start or live migrate as of a certain date. The work around involved setting back the date on VMware ESX server hosts, adversly affecting the date settings on guests. Further the bug surfaced on Microsoft's "patch Tuesday". Patches to a host linux OS are performed using defined methods for the underlying linux distribution. A reboot of a linux KVM host is only required if the kernel is to be updated. Patches are reversible. AuthZ and AuthN VMware ESX introduces a new set of authn and authz rules and meta data to maintain, including users, groups, file and directory users / groups, and associated ACL's. Linux KVM uses existing infrastructure and methods for authn and authz, via linux authentication and file system controls. Network Configuration VMware ESX has it's own methods of network configuration and maintenance to provide host networking and bridged, nat or virtual switched networking for guests. Linux KVM uses standard linux bridge or network interfaces on the host and taptun interfaces to provide bridged or nat devices for guests. Virtual ethernet networking (switch or router) is available via the open source Virtual Distributed Ethernet device. In all cases standard linux OS network configuration is used. Guest Console Access VMware ESX guest consoles are accessible via proprietary windows management tools, a web interface, or vnc. Vnc access requires extra configuration for each guest, and adding users to the VMware ESX server (for ssh port forwarding) or opening the ESX server firewall to non password protected vnc console connections. Linux KVM guest consoles are accessible remotely through vnc. Linux KVM does not require remote root login to access the vnc consoles and we used linux authn/authz to control access to guest consoles for testing. Linux KVM vnc allows connecting the console to a UNIX socket, TCP port, or neither with future dynamic assignment of the console. Direct remote access options include TLS, a per guest password, via x509 certificate, a host based acl, and a reversed connection pushing the console to a specified host. 12
Remote access Remote access to the VMware management console can be done via Windows RDP. Access to linux KVM hosts can be done in many ways, and occurred over SSH during testing. VMware has an extensive GUI management interface which runs on a separate server, and is a separately licensed product with support for a finite number of VMware hosts and clusters. The linux KVM interface runs via the command line, with a set of console command tools available from each running guest's console interface window. Several native GUI and web driven interfaces exist for managing linux KVM hosts and guests. Scripting support VMware ESX has limited scripting support, with most operations designed to take place via the management console. Linux KVM has all of the scripting support offered by the linux OS. An extensible perl script for managing linux KVM guests across a cluster of hosts was built during testing, exceeding the functionality of the VMware command set. Resource monitoring VMware ESX resource monitoring takes place via the management console, and includes reporting on host and guest resource usage. Linux KVM hosts are monitored like any other linux server. Linux KVM guests are linux system processes and their resource usage is monitored as such. Linux KVM monitoring can take place via our current monitoring tool sets (syslog, zenoss, whats up gold, cacti). 13
Performance VMware performance is good but supposedly suffers under heavy IO loads. Linux KVM has exhibited near native performance in all of our testing. Utilizing the bonnie++ benchmark for disk IO VMware and linux KVM guests were comparable for sequential character and block IO, but linux KVM was far ahead in the creation and deletion tests. Linux KVM includes support for para virtual and pass through drivers for increasing network and disk IO performance. Recent development versions of linux KVM tout native gigabit ethernet performance. Bonnie++ benchmark output (++++ means < 500ms): Version 1.03b ------Sequential Output----- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP VMware 496M 37912 85 56537 15 10022 3 41311 91 247546 98 11059.9 31 VMware 496M 32250 75 74378 19 9415 3 37445 89 48146 11 5976.7 29 VMware 496M 39021 95 87659 24 12641 2 38573 83 239168 92 8649.8 19 KVM 496M 47453 96 43353 9 1144 0 42023 80 130318 14 8913.9 55 KVM 496M 50196 97 81835 14 16315 4 43245 73 128881 14 8689.8 28 KVM 496M 50193 97 151666 26 38087 8 43627 69 173857 18 8929.4 53 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP VMware 16 2866 98 +++++ +++ +++++ +++ 3023 98 +++++ +++ 12812 100 VMware 16 2870 96 +++++ +++ +++++ +++ 2974 98 +++++ +++ 12704 99 VMware 16 2875 99 +++++ +++ +++++ +++ 2976 98 +++++ +++ 12503 99 KVM 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ KVM 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ KVM 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 14
Budget Considerations Our initial infrastructure design for virtual server hosts consists of 4 production servers and 1 test server. The servers are already purchased but their cost is included in the numbers below for reference. A 3 year warranty replacement cycle for server hosts is used. VMware's license fees are high. We could purchase servers at a rate of 5.6 every year instead of licensing VMware. Also included are numbers assuming RedHat Enterprise licensing for production linux KVM servers. VMware $3000 / hardware server * 5: $15,000 / 3 years $3000 / host licensing * 4 : $12,000 / year 3 year cost: $51,000 6 year cost: $102,000 Annualized: $17,000 Linux KVM $3000 / hardware server * 5: $15,000 / 3 years $0 / host licensing : $0 / year 3 year cost: $15,000 6 year cost: $30,000 Annualized: $5,000 Linux KVM with RedHat Enterprise support $3000 / hardware server * 5: $15,000 / 3 years $725 / host licensing : $2900 / year 3 year cost: $23,700 6 year cost: $47,400 Annualized: $7,900 Hardware versus VMware licensing: $51,000 VMware option / $3000 per server: 17 servers / 3 years Annualized : 5.6 servers / yr 15