WWW.QUALTECHCONFERENCES.COM Europe s Premier Software Testing Event World Forum Convention Centre, The Hague, Netherlands The Future of Software Testing Best Practices for Testing In Virtual Environments Martin Klaus, Vmware, USA
Best Practices for Testing in Virtual Environments Martin Klaus
Agenda Virtualization Basics Why is Virtualization such a big trend? Taking Advantage of Virtual Labs for Testing and QA Architecting Virtual Labs Best Practices for Testing in Virtual Labs Virtual Labs at VMware Performance Testing Considerations Summary and Next Steps
Virtualization Basics
What is Virtualization? Physical Machine Virtual Machine Virtualization packages hardware + OS + applications into a portable virtual machine package
Key Properties of Virtual Machines Partitioning Run multiple operating systems on one physical machine Divide system resources between virtual machines Isolation Fault and security isolation at the hardware level Advanced resource controls preserve performance Encapsulation Entire state of the virtual machine can be saved to files Move and copy virtual machines as easily as moving and copying files Hardware-Independence Provision or migrate any virtual machine to any similar or different physical server
Physical Hosted Server-based Virtualization Run several operating systems on a single machine. Create shared pools of resources to optimize your infrastructure.
Why is Virtualization such a big trend?
Platform for Business Critical Applications Separate Consolidate Aggregate Automate Liberate CapEx Savings OpEx Savings Business Agility Automation Virtual Infrastructure Virtual Infrastructure Management Management Management Hypervisor Hypervisor Hypervisor Hypervisor Test and Development Server Consolidation Capacity On Demand Self-Managing Datacenter Computing Clouds
Business-Critical Application Momentum >100,000 companies worldwide virtualize their datacenter 54% deploy enterprise applications on virtual infrastructure 46% standardizing on virtual infrastructure 63% plan to virtualize more than 50% of their x86 infrastructure in 3 years In a recent ESG survey, nearly half of the current users consider themselves to be running mission critical applications on virtual machines. ESG Research, December 2007 Test and Development Server Consolidation Capacity On Demand Self-Managing Datacenter Computing Clouds
Virtualizing critical applications in production 92% of our customers are confident that their mission-critical apps run well on VI Percent customers running these applications in production MSFT Exchange 36% MSFT SharePoint 54% MSFT SQL Server 62% ORCL Application Server 31% ORCL Database 32% IBM Domino 37% IBM DB2 20% IBM WebSphere 44% All Databases 42% Source: VMware customer survey, October 2007, sample size 500
Taking Advantage of Virtual Labs for Testing and QA
Common Test Lab Challenges Inefficient provisioning of test environments Setup, teardown, reset and configuration time Process, time and budget dependency on IT
Customer Example: Average Machine Utilization Maintenance Tasks Non-test related tasks such as Imaging, Installing, Cleanup, or Locked for Triage. Testing Tasks Test execution Maintenance Tasks Testing Tasks On Average, test execution is only 10% of the total machine utilization 80% 70% 60% 50% 40% 30% 20% 10% 0% Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Image Machine Install Software Run Tests Await Triage Release for new test 20 Minutes 5 Minutes 20-120 Minutes 1-12 Hours!!
Common Test Lab Challenges Inefficient provisioning of test environments Setup, teardown, reset and configuration time Process, time and budget dependency on IT Constraints of physical hardware Test matrix limited by available lab resources Little asset sharing leads to server sprawl
Customer Example: Physical Test Lab Challenges Resources locked until test run complete Every task must go through imaging and product install No reuse of common configurations Remote access issues Requires hardware specific Images Consumes lots of power and space
Common Test Lab Challenges Inefficient provisioning of test environments Setup, teardown, reset and configuration time Process, time and budget dependency on IT Resource requirements Test matrix limited by available equipment Little asset sharing leads to server sprawl Collaboration of development and testing teams Reproducing environments to troubleshoot problems
Customer Example: Before Virtual Test Lab Before Virtual Test Lab Number of provisioning requests and slow IT response time (days) High temperatures and power consumption of 120 server spread across multiple labs Significant number of machines reaching the end of their life cycle causing unexpected downtime Time and complexity required to manage and maintain system configurations Inability to provide service for remote users without manual intervention
Customer Example: Virtual Test Lab 7 Blade Servers running 20-50 VMs each Supports 120 concurrent lab users in multiple locations Increased lab capacity by an additional 200 virtual machines Improved IT response time from days to minutes with a self-service portal Fewer machines and dramatically reduced hardware replacement cost After Virtualizing the Test Lab
Customer Example: Test Automation Infrastructure developed over 6 years Roughly 100 unique users per day and over 300 per month Support team of 8 full-time engineers Mostly automated GUI testing but also includes programmatic SDK + API tests Averaging roughly 86% increase in testing year over year 80000 70000 60000 50000 40000 30000 20000 10000 0 Test Executions by Release Feb Mar Apr May Jun Jul Aug Sep 06 Release 07 Release 08 Release Annual Release Cycle Launch in Fall
Architecting Virtual Labs
How Virtual Lab Infrastructure Works User selects a multi-machine configuration, clicks deploy. Virtual Lab Portal Virtual machines stored in the image library are deployed on the virtualized server pool Image Storage Library Elapsed Time: 1-2 minutes LAN/SAN QA Engineer Once deployed, user directly interacts with the machines, as if sitting at each console Virtual Infrastructure Virtualized Server Pool Automated Virtual Lab
Virtualized Server Pool Sizing Infrastructure Requirements Server-based virtualization typically achieve consolidation ratios of 10:1 or higher QA and test labs often achieve 25:1 or more Use 4-way servers for higher VM density Basic sizing formula: Number of Virtualized = Servers needed Users * Machines Consolidation Ratio
Storage Sizing Image Library Efficient use of storage through linked clones and delta disks Recommendations Storage need depends on the number of base images + length of linked clones chain Allocate several large VMFS data stores (300GB - 500GB each) for initial VM storage Minimum 200GB disk space for media storage Storage Server A B C A B C A B C A B C WinXP base disk Win2k3 base disk
Test Automation in Virtual Labs Use Test Automation Tools Most vendors have plug-ins and integrations with virtual lab automation products Build your own automation scripts specific to your environment and processes, e.g. automated regression testing after check-in Capture Test Environments Snapshot virtual machines if tests fail linked clones reduce storage overhead Discard virtual machines if tests succeed
Best Practices for Testing in Virtual Labs
Problem: Test Matrix Management Test matrix is becoming too difficult to manage Host operating system, browser, database, application server, localization, JRE, etc Trade-off between test matrix, release schedules and quality must be made
Solution: Using Templates and Configurations Create multi-level templates Level 1: OS Level 2: OS + Patches Level 3: OS + Patches + Apps Tests are downloaded to Level 3 configurations Build VM Configurations from Templates Preserve Gold master images Instantiate templates in multi-tier configurations Image library stores all test environment configurations Selectively deploy, combine or split VM configurations for combinatorial use cases
Problem: Test Resource Collisions Different types of testing cannot be executed simultaneously in same environment Regression and Integration Testing User Acceptance Testing Test Environment Installation and Upgrade Testing
Solution: Clones and Network Fencing Deploy multiple identical environments simultaneously Linked clones and delta disks reduce storage costs Configuration B Clone of A Regression and Integration Testing User Acceptance Testing Configuration A Original Configuration C Clone of A ESX Server Pool Installation and Upgrade Testing
Problem: Modified Test Environments Environment Impact Tests Installation, Upgrade and Patch testing Test Data Management Maintain test data through data destructive test cycles
Solution: Store Application State Snapshots and Revert Points Capture to Library and Checkout of Library Test Execution Pristine Data State Unknown Data State Revert to Snapshot or Checkout Library Configuration
Problem: Not-Reproducible Defects Non-determinism is source of many bugs Race conditions in multithreaded code Application behavior dependent on user input, network, etc Non-reproducible defects get dismissed and shipped Post-mortem debugging Applications don t always crash Post mortem represents a system snapshot at the time of the crash but doesn t capture behavior leading up to the crash
Solution: Record and Replay Record and replay whole VM execution i.e., all threads, all processes, and operating system Instruction exact Recorded session encapsulated in a file User Network Disk
Record and Replay Benefits Instruction exact recording of VM execution Inexpensive enough to use pervasively (won t miss bug) Use go-live to evaluate what-if scenarios Replay bug over and over Records all asynchronous data (user input, network, etc) Minimal execution dilation (won t mask bug) Memory leaks Race conditions Minimal probing effects during debugging Record bug with low overhead Attach debugger while replaying Control/inspect program in usual ways: breakpoints, watch points, single stepping, etc.
Virtual Labs at VMware
How VMware QA uses virtualization Virtual Development Environments VDE contains IDE, compilers, build tools and system paths Switch between VDE to get different version build tools Quickly provisioning VDE for new hires and contractors Virtual Test Environments VTE contains deployment environment and application build Manual and Automated Testing team access VTEs We also use test and build automation tools Development and QA Collaboration Code check-ins trigger automated smoke testing Continuous integration testing against main branch
Virtual Lab Automation in Development and Test 1. Daily build produced by build System Build System 2. VMware Lab Manager SOAP API used to checkout reference platforms, install build, and download and run latest smoke test scripts SQA System Tests Lab Manager QA/ Dev 3. Bugs are sent to dev team and filed in bug tracking system using LiveLinks 4. All results are checked into the Lab Manager Library. Leases used to keep library clean. Fixes 5. Fixes checked into build system for next day s build
Performance Testing Considerations
Performance Expectations for Virtual Infrastructure Expect CPU overhead to be in the order of 5-20% This is what we see consistently in partner benchmarks Expect Storage latency overhead to be negligible ESX Layer adds about 0.2ms per I/O, and is capable of >100,000iops NFS and iscsi protocols often provide sufficient performance, but can have slightly higher CPU overhead VMware ESX Scales Out Well Adding VMs scales well: over 95% scaling factor
Single VM, CPU Intensive Performance VMware ESX 3.x compared to Native SPECcpu results covered by O.Agesen and K.Adams Paper Websphere results published jointly by IBM/VMware SPECjbb results from recent internal measurements
Measuring the Performance of DB Virtualization Throughput Delivered Minimal Overheads
Oracle 10GR2 Swingbench Throughput (txns/sec) Throughput scales with the size of the VM Number of virtual CPUs in Database/Guest
Total System Throughput (ops/s) Total System Throughput (ops/s) Application Tier Scales Better in Virtual Machines Java on Linux Java on VMware Java Virtual Machines 1 2 3 4 5 6 7 8 Java Virtual Machines 1 2 3 4 5 6 7 8 3-Tier client-server Java benchmark measuring num. of business transactions per sec. Intel Xeon 3.2 GHz, 2 GB RAM, VMware ESX 3.0, BEA LiquidVM 1.1, BEA JRockit R27.3, BEA WLS 9.2 MP2, RHEL 4.0. Each VM allocated 1 vcpu and 1 GB vmem. JVM -Xmx=800MB, 135 MB live data.
Summary and Next Steps
Building your own Virtual Test Lab Implement proof of concept prior to broader deployment Get your IT department involved Virtualize one or two QA test lab servers to gain experience Scale out your virtual lab over time Online Resources VMware Web site: http://www.vmware.com/solutions/vla/ White papers, Technical articles, community forums, blogs, customer references Free Tools VMware Converter, VMware ESXi, VMware Server
Conclusion Virtualize your test lab Start by leveraging existing servers and local storage Import physical test configurations as virtual machines Use test automation tools Included automated provisioning and capture of test environments in your tests From 2007 Customer Survey Time is the biggest item saved through virtual lab automation 50% increase in defects found prior to production/shipment